0
$\begingroup$

I'm trying to use Instant NGP on a synthetic image dataset. I have image sequences of a moving object with the following information:

  • I have the object's xyz and the camera's xyz positions in a void.
  • The object can rotate, and I have it's yaw, pitch, and roll values.
  • The object is always centered in the images.

To use Instant NGP, I need to provide camera-to-world transforms for each camera, with the camera rotated to look at the object which is centered around [0,0,0] and is in a yaw = pitch = roll = 0 state. I'm a 2D vision person and I can't quite wrap my head around this, and my latest implmentation (based on this) is close but not right.

My current approach is:

  • Centre the object at [0,0,0]
  • Move the camera to the correct relative position (camera - object)
  • Rotate the camera's position about the origin to account for the rotation of the object (roll then yaw)
  • Rotate the camera's look angle to be pointing at the origin (roll, then yaw, then pitch)
  • Rotate a constant pi/2 roll offset to point the image up.

But there's a lot of signs, angles, and axes to keep track of and I'm getting something wrong. It is entirely possible my definitions used above are wrong due to inconsistent axes labelling.

Any advice on how to do this, or the correct terminology for me to search for, would be appreciated. I'm working in python if there are any helper libraries I should be using.

EDIT TO ADD MORE DETAIL (approach above exapnded): I tried to get it working on 4 known points (a close pass over the object (in front, directly above, behind), and one from far above.) These are pointing in roughly the right direction but not exactly centered, but I'm happy with the relative positions between object and camera.

Cameras used to develop code against, yellow box is an inset view

This is an example sequence of views. Again the relative positions seem correct given the object's orientation, but the cameras are mostly looking in the wrong directions:

Example image sequence Cameras pointing in the wrong direction

Improve this question
$\endgroup$
5
  • $\begingroup$ It sounds like the camera is in a fixed position and the object is also in a fixed position, is this true? If this is true you can use a fairly standard "LookAt" function to generate the view matrix. $\endgroup$
    – pmw1234
    Aug 18 at 14:55
  • $\begingroup$ Welcome to computer graphics stack exchange =). You said, you are close to the solution. Please discribe how your solution works (which rotation order...) and what is different to the correct solution? Maybe some pictures will help to understand where your code fail and how to fix it. Which of your 4 points work? And which don't? $\endgroup$
    – Thomas
    Aug 18 at 19:17
  • $\begingroup$ I remember instant-npg / nerfstudio / original nerf and opencv has pretty different coordinate frame settings (like which is the 'upward-pointing' axis and which is the forward one). I was once bothered by this, and I think the coordinate frame conversion of different models is going to help: github.com/NVlabs/instant-ngp/issues/658 $\endgroup$
    – Enigmatisms
    Aug 19 at 16:11
  • 1
    $\begingroup$ Thanks all - post edited to add more detail and example images. pmw1234 - each "camera" is static, but there are images taken from around the scene (see images). A LookAt function sounds like exactly what I need, I'll search and try. Thomas - added some detail, although my axis labelling is likely wrong/inconsistent which is part of my problem! Enigmatisms - thanks, this looks very useful. Hopefully combining this and a LookAt function I can find a solution. $\endgroup$
    – Thom
    Aug 21 at 9:20
  • $\begingroup$ Update: this post was extremely useful, I think I've got a working solution now. Will confirm and update. $\endgroup$
    – Thom
    Aug 21 at 11:00

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.