I'm writing my own UV Unfold operation (similar to Maya's Optimize UV tool or 3DS Max's Relax Tool). My code works great 90% of the time but in some instances, I'm getting faces and edges that flip on top of each other. This causes the faces to orient in a different direction than intended.
I'm trying to find a good approach to detecting when this flip occurs to either prevent it beforehand or correct it when it does happen. I'm currently testing the dot product of the 2D normal compared with the dot product of the previous normal after I move the uvs. Is this a decent approach? Am I missing another simple algorithm or check that could prevent the overlap?
3D mesh for testing
Results with my algorithm with overlapped faces and edges.
Maya's optimize results.