I'm trying to use Instant NGP on a synthetic image dataset. I have image sequences of a moving object with the following information:
- I have the object's xyz and the camera's xyz positions in a void.
- The object can rotate, and I have it's yaw, pitch, and roll values.
- The object is always centered in the images.
To use Instant NGP, I need to provide camera-to-world transforms for each camera, with the camera rotated to look at the object which is centered around [0,0,0] and is in a yaw = pitch = roll = 0 state. I'm a 2D vision person and I can't quite wrap my head around this, and my latest implmentation (based on this) is close but not right.
My current approach is:
- Centre the object at [0,0,0]
- Move the camera to the correct relative position (camera - object)
- Rotate the camera's position about the origin to account for the rotation of the object (roll then yaw)
- Rotate the camera's look angle to be pointing at the origin (roll, then yaw, then pitch)
- Rotate a constant pi/2 roll offset to point the image up.
But there's a lot of signs, angles, and axes to keep track of and I'm getting something wrong. It is entirely possible my definitions used above are wrong due to inconsistent axes labelling.
Any advice on how to do this, or the correct terminology for me to search for, would be appreciated. I'm working in python if there are any helper libraries I should be using.
EDIT TO ADD MORE DETAIL (approach above exapnded): I tried to get it working on 4 known points (a close pass over the object (in front, directly above, behind), and one from far above.) These are pointing in roughly the right direction but not exactly centered, but I'm happy with the relative positions between object and camera.
This is an example sequence of views. Again the relative positions seem correct given the object's orientation, but the cameras are mostly looking in the wrong directions: