Tesselation sine distortion?

Question

I am experimenting on a simple wave tessellation shader. I ported it from a working vertex shader. When my frequency is higher than 1, I get strange "saw-tooth" artifacting at peaks of the wave. Here are some images with a frequency of 3.

If I keep the frequency lower, it is smooth as butter. Any ideas what the problem could be?

#version 410 core
layout(quads, equal_spacing, ccw) in;
in vec2 teUv[];
out vec2 fUv;
out vec4 fWorld_pos;
uniform mat4 VP;
uniform mat4 M;
uniform sampler2D water_disp;
uniform float time;
const float M_PI = 3.1415926535;

float water_displacement(vec2 pos) {
    const float freq = 3.0; // <-- This is the problematic value.
    float d = length(pos);
    return (0.5 + 0.5 * sin(-M_PI * d * freq + time));
}

void main() {
    vec4 p1 = mix(gl_in[0].gl_Position,
            gl_in[1].gl_Position,
            gl_TessCoord.x);
    vec4 p2 = mix(gl_in[2].gl_Position,
            gl_in[3].gl_Position,
            gl_TessCoord.x);
    vec4 local_pos = mix(p1, p2, gl_TessCoord.y);
    fWorld_pos = M * local_pos;
    fWorld_pos.y += water_displacement(fWorld_pos.xz);

    gl_Position = VP * fWorld_pos;

    vec2 uv1 = mix(teUv[0], teUv[1], gl_TessCoord.x);
    vec2 uv2 = mix(teUv[2], teUv[3], gl_TessCoord.x);
    fUv = mix(uv2, uv1, gl_TessCoord.y);
}

And more examples:

Answer 1

Not enough triangles. The situation is analogous to the sampling theorem that states that you can not reconstruct a signal if your sample frequency is below a certain threshold. Although in this, case you're not attempting to rebuild the signal with a higher order filter so the result is even more dramatic. And it results in interpolation errors.

You can alleviate the problem with less points if your point wavefront follows your waves. So you may want to shift points towards the gradient maximum in other directions than UP.