If the iPhone had LiDAR sensors, it would be possible to generate a 3D point cloud that you could theoretically use to furnish a 3D AR reconstruction of your surroundings. E.g. you could create an AR video chat whereby you see a "hologram" projection of the person you are speaking within your 3D environment.

However, given that LiDAR has yet to be released on iPhone, do you know of an alternative transformation algorithm that may be employed on a single video stream for consistent video depth estimation?

I came across this SIGGRAPH 2020 video depth estimation project which seems to do this pretty reliably. I think this question ultimately falls with the class of SLAM problems. The SIGGRAPH paper seems to be very promising, but ideally, this would be done in realtime (or with a small time lag), which doesn't seem plausible; this is a demo video in case you're curious.

Alternatively, I thought it might be possible using two cameras simultaneously, e.g. the wide and normal, to correct for the camera FOV and optic distortion in each and then use the 3D stereoscopic transform in OpenCV knowing the optical properties of the setup (distance between the cameras, focal length, etc.) to generate a 3D image of the scene with almost as much precision as LiDAR. This Medium article on Stereo 3D reconstruction seems to suggest that such a configuration is possible, although I suppose the small baseline distance between the cameras relative to the object might affect the outcome. I think depth map in Apple's AVFoundation is certainly an alternative; however, the TrueDepth camera system was only introduced with the iPhoneX.

Thanks in advance! I would appreciate it if this could be demonstrated by example :) I cross-posted this question on the Computer Graphics exchange.

  • 1
    $\begingroup$ True, the tiny distance between the cameras will strongly reduce 3D reconstruction precision. Once a feature point appears on the very same location in both pictures there is no way to do proper stereo reconstruction for it. I think the critical distance will be quite small. $\endgroup$ – Isolin May 20 at 17:30
  • $\begingroup$ @Isolin Are you aware of an alternative, pre-TrueDepth, i.e. for models prior to iPhoneX? $\endgroup$ – Sergio Charles May 20 at 17:53
  • 1
    $\begingroup$ No, I have "traditional" photogrammetry experience, but honestly I was never interested in Apple products. If you assume your scene to be static, you can fallback to a progressive approach. At least for close objects the stereo could help a bit. But if stereo would be the ultimate solution to go, no one would ever think of integrating other technologies like structured light or LiDAR :) $\endgroup$ – Isolin May 20 at 20:32
  • $\begingroup$ @Isolin Well said! Yes, I suppose a Stereo solution would be limited to close objects. $\endgroup$ – Sergio Charles May 20 at 21:58
  • 2
    $\begingroup$ Estimating depth from monocular images and videos is a quite hot topic. I id not keep an eye on it for a while but if you are looking for similar projects then just google.com/search?q=depth+from+monocular or google.com/search?q=depth+from+monocular+video. There are many with code and benchmarks available. $\endgroup$ – datjko Jun 1 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.