How can I offset/shrink a triangular polygon in GLSL?

Question

I need to offset all (blue) triangles, each independently of the others, using the vertex-shader. In order to manipulate the triangle as a whole, I've created custom (vec3) attributes for each vertex (red) representing the leftward (purple) and rightward (green) neighboring vertices. From this, I need to derive the orange point, equidistant (in screen space) from both adjoining edges. With three such orange points derived from each triangle, the processed (orange) triangle is passed on to the fragment shader.

Ideally, the triangle will be culled (as in backfacing/ not rendered) if the offsets negate any available space within the triangle, such as in the second triangle in the second picture.

I'm using THREE.BufferGeometry() as my data structure.

Here is a screen shot of the effect I'm aiming for:

Answer 1

Given triangle ▲ABC, we bisect angle ∠BAC with line AD, derived with Angle Bisector Theorem:

BA / BD = CA / CD Point E represents our objective refined position on the desired resulting inset triangle. As it lies upon angle bisector AD, it is equidistant from sides BA & CA, forming identical right triangles ▲AFE & ▲AGE. We can now use Sine for Right Triangles to find the length of AE:

AE = EG / Sin(∠EAG)

That's all the math we need, so let's cook up some GLSL!

We start off with all the typical attributes: position, normal, and transformation matrices, but since the vertex shader only works on a single vertex, we need to add the neighboring vertices as additional attributes. This way, each vertex will find its own "point E", creating the resulting inset triangle. (Note: I don't call them "B" & "C" here, because they are not yet in screen space.)

    attribute vec3 left; //vertex to the left of this vertex
    attribute vec3 right; //vertex to the right of this vertex

Speaking of screen space, I'm also including the aspect ratio of the display, (and making it a uniform, in case the window is resized.)

After preparing varying normals for the fragment shader, and transforming the face into clipping-space, we can get down to the business of applying the above math:

        attribute vec3 left; //vertex to the left of this vertex
        attribute vec3 right; //vertex to the right of this vertex
        uniform float aspect;
        varying vec3 vNormal;
        varying vec2 vUv;

        void main() {
            vNormal = normal;
            vUv = uv;

            mat4 xform= projectionMatrix * modelViewMatrix;
            vec4 A = xform * vec4( position, 1.0 );
            vec4 B = xform * vec4( left, 1.0 );
            vec4 C = xform * vec4( right, 1.0 );

            vec3 CB = C.xyz - B.xyz;
            vec2 BA = B.xy - A.xy;
            vec2 CA = C.xy - A.xy;
            float lengthBA = length(BA);
            float lengthCA = length(CA);
            float ratio = lengthBA / ( lengthBA + lengthCA );
            vec3 D = B.xyz + ratio * CB.xyz;
            vec3 AD = D - A.xyz;
            vec3 bisect = normalize(AD);

            float theta = acos( dot(BA, CA) / (lengthBA * lengthCA) ) / 2.0;
            float AE = 1.0/(sin(theta)*aspect);
            newPos.z += AE/length(AD) * (D.z - A.z);
            newPos.x += bisect.x*AE;
            newPos.y += bisect.y*AE;

            gl_Position = newPos;
        }

This code gives us the results below.

Note, there are a few edge-cases having to do with almost backface-culled triangles being flipped by this process, and I began to address this in code, yet decided to simply avoid these cases for now. Perhaps I'll revisit it when I get this project done.

Answer 2

This can be achieved without trigonometric functions by scaling down the incircle of the triangle.

incircle() computes the incircle of the triangle formed by the vertices A,B,C, it returns center and radius as vec4. The vertices X=A,B,C are then moved inwards by the fraction of their distance to the incircle center (Q-X) which is equal to the ratio of desired margin to incircle radius (m/Q.w).

vec4 incircle(vec3 A, vec3 B, vec3 C) {
    float a = length(B - C), b = length(C - A), c = length(A - B);
    float abc = a + b + c;
    // http://mathworld.wolfram.com/Incenter.html
    vec3 I = (a * A + b * B + c * C) / abc;
    // http://mathworld.wolfram.com/Inradius.html
    float r = 0.5
            * sqrt((-a + b + c) * (a - b + c) * (a + b - c) / abc);
    return vec4(I, r);
}

vec3 A,B,C; // vertices
float m; // margin
vec4 Q = incircle(A,B,C);
A += clamp(m / Q.w, 0.0, 1.0) * (Q.xyz - A);
B += clamp(m / Q.w, 0.0, 1.0) * (Q.xyz - B);
C += clamp(m / Q.w, 0.0, 1.0) * (Q.xyz - C);