When performing the unfolding the way you want, after you have unfolded one triangle in the plane, the next one should be one of the three triangles, adjacent to the unfolded one, i.e. the next triangle to be unfolded shares a common edge with the unfolded one.
Assume you have a triangle $ABC$ in three space with vertices $$A = [A_1, A_2, A_3], \,\, B = [B_1, B_2, B_3],\,\, C = [C_1, C_2, C_3]$$ which is one of the triangles from the 3D mesh.
If $ABC$ is the first triangle you are starting the unfolding with, then:
- calculate $l_{AB} = ||A - B|| = \sqrt{(A_1 - B_1)^2 + (A_2 - B_2)^2 + (A_3 - B_3)^2}$;
- set the 2D vertices $a = [0,0]$ and $b = [l_{AB}, 0]$ as well as $$l_{ab} = l_{AB};
Else if $ABC$ is a next triangle to be unfolded in the plane, i.e. you have already unfolded one of $ABC$'s adjacent (neighboring) triangles, therefore one of $ABC$'s edges is already unfolded in the plane. Say that this edge is $AB$ and it is unfolded in the plane as the edge with vertices $a = [a_1, a_2]$ and $b = [b_1, b_2]$, where $l_{ab} = ||a-b|| = \sqrt{(a_1 - b_1)^2 + (a_2 - b_2)^2} = ||A-B|| = l_{AB}$.
Either way, for the triangle $ABC$ you already have the edge $AB$ unfolded as $ab$ in the plane, so that $l_{ab} = l_{AB}$ (i.e. you already know $a, \, b$ and $l_{ab}$). The question is, how do you find the coordinates of the third vertex $c = [c_1, c_2]$ so that $l_{bc} = l_{BC}$ and $l_{ca} = l_{CA}$? We impose the latter two conditions because two triangles that have pairwise equal edge-lengths are congruent (isometric, i.e. identical copies of each other).
For your information, I will write vectors like $B - A = [B_1 - A_1, \, B_2 - A_2, \, B_3 - A_3]$ and $C - A = [C_1 - A_1, \, C_2 - A_2, \, C_3 - A_3]$ and $b-a = [b_1 - a_1, b_2-a_2]$ and will use dot product $\cdot$ and cross product $\times$ between vectors. You already know what $||B-A|| = l_{AB}$ and $||C-A|| = l_{CA}$ are and that $l_{AB} = l_{ab} = \|b-a\|$.
Calculate $$s = \frac{\|(B-A)\times(C-A)\|}{l_{ab}^2}$$
Calculate $$c = \frac{(B-A)\cdot(C-A)}{l_{ab}^2}$$
Calculate both pairs of coordinates
\begin{align}
&c'_1 = a_1 + c(b_1-a_1) - s(b_2 - a_2)\\
&c'_2 = a_2 + c(b_2-a_2) + s(b_1 - a_1)
\end{align}
and
\begin{align}
&c''_1 = a_1 + c(b_1-a_1) + s(b_2 - a_2)\\
&c''_2 = a_2 + c(b_2-a_2) - s(b_1 - a_1)
\end{align}
You have two candidate points $c' = [c'_1, c'_2]$ and $c'' = [c''_1, c''_2]$
Perform a check whether $c'$ is on the same side of the edge $ab$ as the triangle unfolded right before $ABC$.
If it is not, then $c = c'$
else if it is, then $c = c''$
As a result of this procedure, you have constructed the planar unfolding $abc$ of triangle $ABC$ and the next triangle from the 3D mesh to be unfolded (if any), should be the one that shares either edge $BC$ or $CA$ with $ABC$. Either way, for that next triangle, you already know the unfolding $bc$ of the edge $BC$ and the unfolding $ca$ of $CA$ (whichever is relevant), so you can apply the same procedure for that next triangle. And so on and so forth, until you have unfolded all triangles from the mesh.
Have in mind that there are 3D meshes that do not posses unfolding that do not self overlap, no matter what you do. For such meshes there no greedy algorithms or any other algorithm that can do it.
