2
$\begingroup$

Using deep learning to estimate surface normals from monocular RGB images is a common task. The resulting image generally looks like this:

RBG to surface normal map

My question is how can I use this map to actually draw a normal vector (arrow) for any given point on the image? For example, I want to draw a line on the countertop that points up.

Most relevant tidbit I found on the topic: "The RGB color channels (red, green, and blue) in a normal map correspond to the respective X, Y, and Z coordinates of surface normals."

But I don't get what this means. The values are standard RGB, i.e., for each coordinate, you get three numbers in range [0, 255]. Just by how color changes between surfaces, it is clearly related to normals (obviously) but when I tried quiver plots using matlab I couldn't get reasonable results.

TLDR: What is the relation between surface normal maps and actual normal vectors? How can I draw a 3d line given a point on the image which represents the normal vector perpendicular to that surface at that point?

$\endgroup$
2
$\begingroup$

Typically the normal vector, which has XYZ components in [−1, 1] is linearly mapped to the RGB range [0, 255]. You can retrieve the normal vector by reversing this mapping.

For completeness, the encoding is:
$$ r = \text{round}((0.5x + 0.5) \cdot 255) \\ g = \text{round}((0.5y + 0.5) \cdot 255) \\ b = \text{round}((0.5z + 0.5) \cdot 255) $$

Decoding is: $$ x = (r/255) \cdot 2 - 1 \\ y = (g/255) \cdot 2 - 1 \\ z = (b/255) \cdot 2 - 1 $$

$\endgroup$
2
  • $\begingroup$ Thank you. Is there a convention as to which channel corresponds to which axis? e.g., R=X, G=Y, B=Z etc $\endgroup$ Nov 11 '20 at 21:11
  • $\begingroup$ Yes always R=X, G=Y, B=Z. There are different conventions about which ways the axes point, though, such as Y up versus Z up. $\endgroup$ Nov 11 '20 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.