I am trying to write a ray tracer to render boxes that are arbitrarily rotated, i.e. not necessarily axis aligned. While I am reasonably comfortable ray tracing axis-aligned bounding boxes (AABB), I don't know how to handle non-axis aligned objects. The geometry becomes quite difficult. I've looked through a textbook (Ray Tracing from the Ground Up) but couldn't find an answer.
How should I do it? Detailed suggestions would be greatly appreciated.
You can assign a coordinate system to each nAABB in such a way that the nAABB becomes an AABB in its own coordinate system. We call this a local coordinate system.
I assume rays are expressed in a world or global coordinate system. In order to test an nAABB for intersection, one first needs to apply the world-to-local transformation on the ray (origin and direction). This way we obtain a transformed ray which we can intersect with an AABB. Once an intersection point is found, we need to transform this back to world space via the inverse transformation (i.e. local-to-world transformation).
In practice an nAABB is thus expressed as an AABB with a pair of world-to-local and local-to-world transformation matrices.
If one applies a scaling ($S$) or translation ($T$) on an AABB, one obtains another AABB. If one applies a rotation ($R$) on an AABB, one obtains a nAABB. Typically the corresponding transformation matrices are multiplied as follows $T * R * S$. Thus translation is applied after rotation. Since the $R$ component must be included in the world-to-local transformation, you need to include the $T$ component as well. The $S$ component can be included or directly applied to a unit AABB centered at the origin of the world-coordinate system. Including a $T$ component requires your world-to-local and local-to-world transformation matrices to be of size $4$x$4$ (expressed in homogeneous coordinates).
Here you can find C++ code to construct all the $4$x$4$ world-to-local matrices and their inverses you probably need.
Note that this approach works for other geometrical objects as well (if the transformations involve scaling, rotation and translation).
