I am making a Software renderer, and I noticed that there is a screen shaking like effect happening. shown in the first video:
I tried doing filling without clipping against the small square, and the result is much better, yet, there is some sort of shaking on the edges as shown in the second video
Finally I tried drawing the clipped triangles outline, and the shaking is more apparent as shown in the third video
My initial assumption was the line drawing was not accurate enough, so I tried both DDA and Bresenham line drawing without any success.
Additionally, I tried different triangulation schemes, I noticed that when I increase the number of created triangles (such as when I triangulate a square using the center as a new vertex instead of using one of the corners) the shaking gets worse. That led me to believe to the issue is related to shared edges in the triangles, and indeed in my triangle filling algorithm, I tried to remove the shared edge and got a bit better results (but it introduced different artifacts, some pixels were missing): https://git.sr.ht/~aliabdulkareem/SoftwareRenderer/tree/FixTrianglulation/item/src/drawing.cpp#L450
That did not however remove it entirely (I believe I should remove the shared edges from the triangulation phase as well, not entirely sure how to do that, one idea is when I triangulate the triangles after the first one, I always more pixel in the direction of the two newly formed edges)
Also it does not explain entirely the second video shaking when there was a shared edge in the diagonal of a square, but it was fine, while the square edges were shaking despite not sharing an edge.
I am a bit puzzled as how to debug this as I can at least think of three different reasons:
1- Triangulation is wrong
2- Floating point inaccuracy
3- Aliasing (I heard that in animations, aliasing becomes very clear, not sure if that is the case)