# How to calculate vertex normals on a mesh with non-planar polygons

If I have a mesh consisting of polygons that are not necessarily triangles and not necessarily planar. As answered in the previous question I asked, there's no correct answer to calculating normals for such polygons. So I'm wondering instead no matter how wrong, what are some of the strategies used in 3d modeling to calculate the vertex normals for such a mesh, specifically for smooth shading?

• What is the surface of such a non-planar "polygon"? There are infinitely many surfaces that can pass through the vertices of such a "polygon". If you define the surface then you can compute the normal based on the partial derivatives. – lightxbulb Mar 18 at 22:30
• @lightxbulb I would like to further specify that I'm referring to simple polygons. To my knowledge in 3d modeling the non-planar polygon surface is visualized by triangulating the polygon. So I guess the surface is that of individual triangles the polygon consists of. – Lenny White Mar 18 at 23:08
• I unfortunately can't use derivatives to calculate the normals. I need to calculate the surface normals on GPU using triangle interpolation (with OpenGL), and for that I need to use vertex normals. – Lenny White Mar 18 at 23:18
• There are more than one way to triangulate a polygon. When it is non-planar the different triangulations result in different surfaces. So first you have to specify what a non-planar polygon means (e.g. specify its surface). Only then can one argue about normals. – lightxbulb Mar 19 at 8:11
• @LennyWhite: Q&A sites are not appropriate for questions of the form "give me a list of all the ways to do X". We generally prefer questions that are more specific in nature. – Nicol Bolas Mar 19 at 15:46

If you're interested in vertex normals specifically, there's an easy answer even for non-planar polygons that avoids the question of defining what the exact surface is: for each vertex, calculate the normal of the plane formed by the two edges entering and leaving that vertex.

More formally, given vertices $$\mathbf{v}_1, \mathbf{v}_2, \ldots \mathbf{v}_n$$ with counterclockwise winding, define the normal at the $$i$$th vertex as: $$\mathbf{n}_i = (\mathbf{v}_i - \mathbf{v}_{i-1}) \times (\mathbf{v}_{i+1} - \mathbf{v}_i)$$ (where the indices wrap around).

You can then proceed to accumulate normals calculated this way from all the faces that share a given vertex, as usual for smooth shading.

As has been discussed, there are multiple ways to define a smooth surface corresponding to a non-planar polygon, but any reasonable way of defining such a surface must converge to the normals as defined here near the vertices, or else the surface can not both be smooth (locally flat) and meet the straight-line edges between the vertices.

A caveat to this approach, though, is that it won't define a normal for collinear vertices (where the entering and exiting edges are parallel), since the cross product goes to zero there. If this is a problem, it might work to patch up such vertices by interpolating normals to them from the surrounding non-collinear vertices.

• Thank you so much, this is perfect! Could I ask if there's any literature describing this method so I can read up more on why this works? – Lenny White Mar 19 at 17:44
• Or generally I would like to find more on this topic, but I don't know to google it. – Lenny White Mar 19 at 18:02
• Sorry, I don't really know if there's any literature on this; I'm just going off geometric intuition. – Nathan Reed Mar 19 at 18:10
• I see, appreciate it anyway! Can I ask "or else the surface can not both be smooth (locally flat) and meet the straight-line edges between the vertices". This is a bit too advanced for me. Could you explain what this means in simpler terms? – Lenny White Mar 19 at 18:24
• If the surface is smooth, then when you zoom in and look at any very small region of it, that region should appear fairly flat (this is basically what it means to be smooth). And the surface must conform to the given vertices and edges, whatever else it does in the middle. So, if you zoom in on a small region near an edge, the surface there must run parallel to the edge so that it can meet the edge. And if you zoom in on a vertex, the surface has to be parallel to both edges around that vertex. That completely determines the orientation of the surface in that small region. – Nathan Reed Mar 19 at 22:27

for smooth shading you usually store the normal vectors per vertex and not per face (flat shading). The normal vector on a specific point on a face will then be calculated by interpolating the normal vectors of the vertices (corners) of the face.

For example when having a triangle: you would use an interpolation which uses the barycentric coordinates of three normal vectors like discribed here.

For non planar polygons, of cause you don't have a specific defined surface. Therefore it depend on you, which kind of interpolation you are using. But it should interpolate with respect to the corners distance.

And don't forget to normalize the vector at the end!

• Unfortunately this was not what I was wondering about. It's given that we need to calculate the vertex normals and then these are interpolated across the triangle's surface. The question is about how to calculate the vertex normals in the first place. This involves taking the normalized average of all the faces it belongs to, if the all faces are triangles. However what I'm wondering about is what are the ways to handle the case where the faces are non-planar polygons with more than 3 vertices. – Lenny White Mar 18 at 18:38
• Maybe you can subdivide the non planar quads/whatever into triangles and use your approach? Or doing something like mesh relaxation that will converge to nice planar results? – Felipe Gutierrez Mar 19 at 14:59
• @FelipeGutierrez Making the polygons planar wouldn't work for me unfortunately. I'm working on a simple 3d modeling application. And I need to be able to have non-planar polygons. Triangulating would work, but at first glance at least it seems like a lot of headache to implement this if I need to be able to work with a dynamic mesh. – Lenny White Mar 19 at 16:51