0
$\begingroup$

I want to insert a logo into an RGB image with proper 3D transformations. I can get the estimated pixel normals of this image using the code here: https://github.com/yindaz/surface_normal. Pixel normals are encoded into RGB channels as described as follows:

The estimated normal will be saved in 16-bit PNG format, where 0-65535 in R,G,B channel correspond to [-1, 1] for the X,Y,Z component of the normal vector. We use the camera coordinates defined as - X points to the camera right, Y points to the camera forward, and Z points to the camera upward. For example, right facing wall are very red, floor are very blue, and you rarely see green as it's parallel to the camera viewing direction.

How can I transform the 2D logo to put it onto a planar surface in the image? I have tried to calculate the rotation angles of the normal to rotate it so that it has the same direction as the camera normal, and rotate the logo according to these angles, but I have no success so far.

As far as I understand from the quote above, normal of the untransformed logo should be $(x:0, y:-1, z:0)$ and let the pixel normal be $(x:a, y:b, z:c)$ with magnitude $M$. I rotate the logo around $X$ by $arccos(a/M)*180/pi$, around $Y$ by $arccos(b/M)*180/pi$, around $Z$ by $arccos(c/M)*180/pi$, and if $b < 0$ rotate around $Y$ by an additional 180 degrees.

Is there anything reasonable in these? Is it possible that I can do this using the estimated normals?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ You need to search for "Homography". Estimate a homography that places the image in the scene based on the normals and transform your image by the homography. $\endgroup$
    – Wyck
    Jan 31, 2018 at 15:35
  • $\begingroup$ So it is possible to estimate a homography using normals, right? I have found homography estimation using two images or through finding vanishing points in a single image so far, but not using normals. Is there a specific term for this that I should search for? $\endgroup$
    – groove
    Feb 1, 2018 at 5:07
  • $\begingroup$ I assume the normals help you estimate depth so that you get 3d points. You don't need that if the surface is planar though. Just identify the four corners of the rectangle that you want to map in the target image and use the homography. $\endgroup$
    – Wyck
    Feb 2, 2018 at 2:41
  • $\begingroup$ This would work only if there is a rectangle and it can be found, right? I need to find areas with no semantically significant content, such as empty walls, and insert a logo there. I couldn't succeed using corner detection-based methods or homography estimation using parallel lines. Therefore I though surface normals can be useful since there is a normal for each pixel. $\endgroup$
    – groove
    Feb 5, 2018 at 13:03
  • $\begingroup$ Normals are as useful as derivatives in Calculus. Much like how you can integrate derivatives to get the original function (+/- a constant), you can do something similar to your normals to get the original surface (+/- a global translation). But knowing the geometry of the surface doesn't describe WHERE you want to put your image on that surface. I feel like I still don't understand how you want to describe where the image goes. But I also feel like you can bypass the problem if you use the homography. $\endgroup$
    – Wyck
    Feb 5, 2018 at 14:18

1 Answer 1

Reset to default
-1
$\begingroup$

Maybe you should try to find Virtual Tangent Plane like in Surface Normal In the Wild

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.