enable / disable transform feedback within glsl for specific primitives

Question

I am using the tessellation shader stage to increase the density of triangles close to camera. Thereby when the camera is too close to a triangle, so that the maximum tessellation is not high enough, it should store the tessellated parts to the transform feedback buffer. Then, in the next render pass, these tessellated parts should be tessellated again to ensure a good triangle to pixel ratio.

To achieve that, the idea is to only render the well tessellated triangles, and only store the "not enough tessellated" triangles to the transform feedback buffer AND NOT render them.

As far as I understood, this behavior is not possible, because all vertices which pass the final Vertex Processing shader stage will be stored to the feedback buffer.

So first: Is there a way to enable / disable the transform feedback writing inside GLSL?

second: And also the other way around: stop rendering but writing to the buffer?

for the last case (stop rendering but writing) the fragment shader could discard these triangles... A better way would be to stop rasterize them... (performance)

Answer 1

What you want isn't really doable. While OpenGL 4.x allows GS's to output to multiple streams of primitives (and to collect them in separate feedback buffers), multi-stream output is restricted to point primitive generation. So you couldn't really render triangles with them later (well, you could, but it would require a full GPU/CPU sync to read the number of primitives written so that you can divide it by 3 and render triangles).

That having been said, this does not seem like a good idea from a performance perspective anyway. Implementations of tessellation are required to offer a minimum of 64 tessellation levels. In the triangle abstract primitive, using 64 for each tessellation level would generate thousands of triangles for that triangle. If that is insufficient tessellation... it's probably best to live with it.

The cost of doing a feedback operation of so many triangles back into the tessellation system would be exceedingly expensive. Especially considering that you need to communicate which triangles that need more tessellation border triangles that don't, so that you can provide the correct tessellation levels to ensure connectivity.

If you truly need REYES-levels of tessellation (where you ensure each triangle takes up a specific pixel size), you may need a compute shader or more likely just use the CPU.