Given a picture of a unit square at an arbitrary angle and position, how can I reconstruct the distance and orientation?

Question

To simplify, assume faces and corners of the square are labeled and the camera lens if whatever makes this simplest (a 90 degree field of view projection?)

The application: using a known and labeled unit square to assist an AR app.

For the case of a perfectly centered rotated on one axis, we can use the estimated distance from the axis the square was rotated around plus the ratio of sizes of the distorted sides.

But for an arbitrary view, the problem becomes much much harder.

Links to the general case of this problem in projective geometry also appreciated

Answer 1

This problem is known as camera calibration. The goal is estimating the various parameters of a "camera":

• The intrinsic parameters: Things like focal length, distortion of the lens, sensor parameters etc.
• Extrinsic parameters: Orientation of position of the camera.

To robustly determine those from an image that contains labelled points you usually need more than one image. A typical way to do this is recording many images of a chessboard pattern. But as soon as you have determined all the intrinsic parameters, you could save those for future applications as they do not have to be estimated over and over again.

I recommend reading the introduction to camera calibration of openCV:

https://docs.opencv.org/3.1.0/dc/dbb/tutorial_py_calibration.html

https://docs.opencv.org/3.1.0/d9/db7/tutorial_py_table_of_contents_calib3d.html