Projective texturing from many images

Question

Problem: I'm trying to dynamically texture a mesh of the real world generated by the Hololens over time using photos also captured from the Hololens, perhaps one photo per second. Each of the photos includes positional information of the Hololens from the time when the photo was taken, which allows me to project the images onto the existing mesh and figure out the UV mapping for each image.

In case of a single image, this seems fairly simple, as it is just standard projective texturing of the mesh. But I will be capturing a potentially endless stream of images over a span of time which will be taken from different positions and orientations. For acceptable performance, I cannot simply store and iterate through all photos and do projective texturing for each.

How can I perhaps generate a texture atlas & accompanying UV mapping for the real world mesh and update it after each image?

Answer 1

Apologies in advance for the poor quality of this answer, but this sounds a little like what an ex-colleague was doing as part of his PhD. The "Free-Viewpoint video" papers listed at the bottom of https://www.researchgate.net/profile/James_Imber might be useful, or at least a starting point to find related work.

Answer 2

I've not done this, but it seems like you could keep a store of a single image and update it with each new image you receive. For example, you capture the first image and project it onto the scene. It likely won't cover the entire scene.

When you capture the next image, assuming it's not from exactly the same place as the previous, you can combine it with the first image by mixing them 50/50. When you get the next image, you want to mix the existing image at 66.666% and the new image at 33.333%, etc.

So I think you'd need to keep the combined image, and a second image that is the count of how many images have covered each pixel. Then:

for each pixel in the output that the new projection covers {
    existing_pixel = current pixel in the existing image
    num_pixels_mixed = current pixel in the image containing the count of images
    new_pixel = current pixel in the newly acquired image
    mix_amount = num_pixels_mixed / (num_pixels_mixed + 1)
    resulting_pixel = mix_amount * existing_pixel + (1 - mix_amount) * new_pixel
    increment the current pixel in the image containing the count of images
}