I somehow understand that it should not work, as quadratic bending and linear fragment interpolation between two vertices is different,
Indeed. If you want to non-linearly distort a mesh object, then you have to sample the nonlinear function frequently to get a reasonable linear approximation to it — that is, you must not have very large or long triangles.
If that were the only problem, then your mesh would wobble weirdly but not be full of gaps. The gaps are a different, but related problem:
but I implemented a greedy mesh optimization, that same faces are merged to one huge mesh.
The greedy meshing algorithm is known to produce “T-junctions” — points where an edge of one quad (pair of triangles) meets the vertices of two (or more) others.
| Quad 1 |
|V1 V2 |V3
| | |
| Quad 2 | Quad 3 |
Under ordinary circumstances, this merely produces a few pixel-sized specks in the rendering, because the rasterization of the edge of quad 1 doesn't quite line up with the rasterizations of the other two.
However, when you introduce a non-linear transformation, then those specks open up into big gaps, because the non-linear transformation moved vertex V2 so that it is not on the line between V1 and V2 at all. Those are the gaps you are seeing.
To avoid these gaps, you must generate a mesh which contains no T-junctions — every edge of every triangle must exactly meet (share two vertices with) a neighboring triangle. (This is true of most meshes created by 3D modeling software, except where an object is intentionally incomplete, such as leaving off the bottom of a tree's trunk.)
The simplest way to achieve this is to remove the greedy mesh optimization, and generate 2 triangles for every block face. Conveniently, this will also ensure that the visual effect of the distortion is consistently sampled.
If you still want to optimize the mesh, you will have to use a different algorithm, which avoids T-junctions — it cannot simply generate one quad for a large area and must instead take neighboring triangles into account. Once the algorithm has decided to put a vertex at a certain corner, all other surfaces meeting that corner must also have a vertex there too rather than an edge passing by. This is a nontrivial problem and I'm not aware of any articles describing a complete algorithm for it, nor do I have such an algorithm in code form. (Perhaps one day I'll write one.)