I have made use of degenerate triangles when rendering a mesh. While the degenerate triangles are not rasterized when using GL_FILL polygon mode, they are when I do mesh debugging with GL_LINE mode.
Why is this the case? Are there any options to prevent rasterization in GL_LINE mode?
I assume you're talking about triangles with two vertices equal, and a third vertex at a different point. They don't appear in filled polygon mode because such triangles have zero area and don't cover any pixel centers. It's not that there's a special rule "degenerate triangles are dropped", it's that they naturally don't generate any fragments when rasterized. (GPUs may still detect and drop degenerate triangles early in the pipeline as an optimization.)
But when drawn as lines, there is no reason why the lines would be dropped. The rasterization rules for lines always produce a line of the specified width in pixels, and when you draw the triangle's edges as lines, if the edges are nonzero length then they will show up.
There is no mode to control this that I'm aware of. There used to be a thing called "edge flags" that could control which edges of each triangle were rasterized or not, but that was deprecated from GPU APIs long ago.
For other options, you could try to use primitive restart to avoid needing degenerate triangles at all, or you could create a separate index buffer for wireframe rendering that doesn't include the degenerate triangles.
