What you do is you load triangles into memory from disk based on what has been hit previously. You can begin with triangles in colse proximity first. The reasoning is that in one area the rays are likely to hit same triangles repeatedly. And eventually you will be somewhat efficient. (For this reason it is a good idea to cache last hit triangle in occlusion tracing that dont care of order)
Second you store the triangles in a spatial tree that allows you to do quick searching from the disk, to renew what portions you are having in memory by proxilimity. So load only branches that will be in the way of the ray. If its somekind of voxel tree like a octree you can even sort secondary rays and solve them by coherence. A BSP tree is also somewhat good at prruning areas.
There are cases where this fails but its reasonably efficient in most scene buckets if your not renderimg noise...