I am interested in a scientific application in which ray tracing is necessary, but also where the light itself alters the geometry of the surface mesh (imagine perhaps that the beams of light have enough energy to damage the surface). Right now I am writing everything from scratch and not using any existing libraries, and thus my (obviously naive) implementation is extremely slow. As I understand it, this boils down to clever methods by which to update the BVH, interleaved with each rendering step.
Some slight elaboration on the problem: when a ray of light hits a surface, we consider two possibilities: either it reflects off the surface (as is typical in a ray tracer), or it impacts the surface and essentially burns a hole into the surface. This alters the geometry of the mesh and affects any future light rays, and thus forces an update of the boundary volume hierarchy. The surface deformation is usually local but can be global-scale, for example if the hole is large enough it may cause the structure to collapse or break off. Thus ray tracing affects the geometry "in real time", but simultaneously geometry is necessary to perform ray tracing at each step as well.
While I'm interested in the algorithmic problem, what I'm more interested in is whether or not there is a graphics library that is well suited to this task, so that I don't have to re-invent the wheel, or dive too deep into the literature. I am quite a beginner when it comes to available software libraries so please forgive me.