Problem: I'm trying to dynamically texture a mesh of the real world generated by the Hololens over time using photos also captured from the Hololens, perhaps one photo per second. Each of the photos includes positional information of the Hololens from the time when the photo was taken, which allows me to project the images onto the existing mesh and figure out the UV mapping for each image.
In case of a single image, this seems fairly simple, as it is just standard projective texturing of the mesh. But I will be capturing a potentially endless stream of images over a span of time which will be taken from different positions and orientations. For acceptable performance, I cannot simply store and iterate through all photos and do projective texturing for each.
How can I perhaps generate a texture atlas & accompanying UV mapping for the real world mesh and update it after each image?