I have written a software rasterizer, and have encountered a (literal) edge case.
At the silhouette edge of a typical object, you have two triangles—the front face and the back face. They are not coplanar, but they of course share that single edge. Say the pixel center / subpixel sample happens to lie very close to this edge—so close that the depth value computed for both triangles is actually exactly, bit-for-bit, identical.
Without special handling for this case, sometimes the "wrong" fragment will get through. For example, with the
< depth function, if the backface renders first, then the depth test for the frontface will fail and the backface will show through. If the frontface renders first, then the depth test for the backface will fail and the frontface will show. The former case produces the wrong image—sometimes quite objectionably so.
I have thought about hacks to fix this like trying to handle the "equals" case explicitly, depending upon the depth function and whether the fragment came from a front face, but it's not clear, especially for the
≥ depth functions, what the "right" answer is. Still, it's about the only thing I can think of trying.
This is basically Z-fighting, but unlike the usual Z-fighting graphics newbies encounter when they don't set the clipping planes right or whatever (and which makes this issue almost ungoogleable), existing hardware definitely does provide some kind of solution to handle this particular case, because this problem does not occur on real GPUs.
So my question is—how is this handled by real GPUs? If that's unanswerable, then how can it be handled consistently?