When to apply backface culling, depending on the ray and material type?

Question

I am currently implementing a ray tracer, which supports reflection and refraction. I have the following types of rays:

• camera rays
• shadow rays
• reflection rays
• refraction rays

I have the following types of materials:

• diffuse (opaque, do not reflect)
• purely reflective (opaque, but can reflect)
• reflective and refractive (semi-transparent objects such as bodies of water, which can reflect part of the light ray and refract the rest).

For which combinations of ray type and material type of object being hit should I perform backface culling, when determining the hit point of a ray on a triangle?

EDIT: From what I stated above, it might seem like my renderer is physically accurate, but it really isn't all that much. Here are some implementation details I failed to mention:

• I calculate the surface and vertex normals in runtime, when parsing the scene. I don't read them from the scene.
• By backface culling, I implied the check if the dot product of the incident ray and the surface normal is greater than or equal to zero. Right now, as long as it's not a shadow ray and the dot product is greater than or equal to zero, I return that there is no intersection.
• The different types of rays are used mainly for debugging purposes so far. The shadow rays are an exception and I use them as stated in the previous point.
• the non-reflective diffuse material is used to represent a material, which absorbs light. When a ray intersects an object of such a material, I calculate the contribution of each light.
• the reflective material causes the incident ray to be reflected. The final color is the color, returned by the reflected ray, multiplied by the albedo of the reflective material, in order to avoid total reflection (perfectly mirrored surface).
• the refractive material causes either a reflection ray, or a reflection and a refraction ray to be traced (based on Snell's law) and the final color is calculated using the fresnel formula.

Answer 1

Consider two cases:

You usually don't cull those back faces in camera space (faces are facing away from the camera). This will lead to incorrect will lead to incorrect indirect lighting and some times ray termination, if you are not using two-sided BSDF (check Mitsuba renderer, two-sided BSDF). For example, you are rendering Cornell Box scene, and the back faces of the boxes are removed. Your ray penetrates into the boxes and queries the BSDF with 0 path throughput (since the BSDF is one-sided), and it will be terminated immediately.

However, if you opt for extreme performance, and slight bias for indirect illumination is acceptable, then back face culling (for opaque objects) can be applied.

---

For back face culling in ray space, i.e. we check whether the ray incident direction $\pmb{w}$ and surface normal (consider simple cases, geometry normal) $\pmb{n}$ can have a negative dot product, if not, the surface is culled. So:

• Reflective material (purely reflective (opaque, but can reflect)): If you are hitting surfaces that has normals pointing inside, and if you are not using two-sided BSDF, then usually the BSDF model will set the path throughput to be 0, by clipping the foreshortening term by max(0, dot(w, n)), so the radiance returned will be zero. If it is visible, it will be some pleasant visual artifact which you want to avoid. Normally, you should fix these normals first (flip the direction during modeling). However, if the scene is too complex to edit, you can either use two-sided BSDF, or cull these back facing surfaces (but might lead to other artifacts, like a hole on the object).
• Material that does not reflect (diffuse (opaque, do not reflect)): ... Emmm, what's for? If there is a surface that does not reflect light, there are mainly two cases this is useful: (1) you place it here intentionally to absorb light (and terminate the path), so why would you cull it? (2) this is an emitter (sometimes we only want emitting surfaces to emit radiance), then why would you cull it?
• Refractive material: no, instead, you need the orientation of the normal to judge whether your ray is penetrating into or out-of the object, since in the refraction cases, index-of-refraction on the two sides of the medium interface is determined this way. So, no, it is undesirable to remove transparent surfaces.

There are indeed some cases you should cull a surface (not only the back facing surfaces), but rare. An example is the scattering medium boundary: we sometimes use a mesh to represent the boundary of a volumetric medium, but we don't want the mesh to affect the ray direction and radiance. Therefore, the culling can be applied to these surfaces, during visibility testing.

DirectX and HLSL do support culling: there are enumerations like RAY_FLAG_CULL_BACK_FACING_TRIANGLES, RAY_FLAG_CULL_FRONT_FACING_TRIANGLES for function TraceRay, but so far I don't find them useful. I could say this is because ray tracing is fundamentally different from rasterization in whether the shading process depends on the concept of ray. If I recall it correctly, since ray-space culling cannot be pre-processed (for camera space culling, it is easy to know what to cull, given the scene), it is computed on-the-fly (by checking the ordering of vertices) and the hit on the culled faces are simply ignored. So basically, you are not alleviating the workload for intersection acceleration structure.