(Crossposting from here: https://softwareengineering.stackexchange.com/questions/399331/fuse-3d-points-in-bundel-adjustment)

I'm actually implementing my own Pose-Estimation/- and -Refinement pipeline. For this purpose I use one moving mono-camera. Then I take the consecutive images to estimate the pose and triangulate the points (nothing special). In the last step I refine the poses and 3D-points with a bundle adjustment approach.

Generating 3D points with triangulation from consecutive image pairs will give me multiple estimations for one real-world 3D point. In fact, all the estimations refer to the same point. For my understanding, these estimations of the same 3D-points have to be fused in some way. Otherwise the poses were not linked anymore through a common point (see also image below). Further, looking at the equation for the re-projection error in different publications:

Core equation Bundle Adjustment

turns out, that 3D point (vector a) is only related to j and not to the cameraindex i.

Stolen from: SBA: A Software Package for Generic Sparse Bundle Adjustment, MANOLIS I. A. LOURAKIS and ANTONIS A. ARGYROS

Do I understand that right or do I have to use a different set of 3D points for each camera view? Suppose I've to merge the 3D points, is there any preferable strategy?

Thanks in advance!

I know, there are already countless implementations for BA. I want to use it for further development...

  • $\begingroup$ I'm voting to close this question as off-topic because it was cross posted on another site and answered there already. $\endgroup$ – Catija Oct 11 '19 at 15:28
  • $\begingroup$ So, everything works now as expected and the results seem to be correct. I took the average of all 3D-Point belonging to one physical point. Therefore this question should be considered as answered. Just a point for further investigation: Taking the average may not be a really robust way. It would be useful to implement something like an outlier control. But this was not part of the question at core. Please consider this as the answer. I can't mark it as an answer anymore... $\endgroup$ – MattDom Oct 12 '19 at 22:01