# Fundamental drawbacks of rasterization over ray tracing

Are there any fundamental drawbacks/limitations of rasterization as compared to ray-tracing? When I search on the internet, I get vague explanations like rasterization does not provide lighting effects as good as ray-tracing. It means that rasterization can do it, but not as good as ray-tracing.

However, I'm looking for some cases which rasterization cannot handle at all, but ray-tracing can. I thought rasterization may fail to handle reflections from mirror or water surfaces, but I found out that there are techniques to handle these in rasterization.

It is generally not so much about the fact that rasterisation couldn't do the things. It probably could, but it would most likely need several rendering passes and all kinds of dirty tricks to accomplish things that depend on nonlocal effects.

Its simply easier to think in terms of allong path of a ray than per layer of total effect. Meaning its also easier to implement. Tracing is also significantly faster than multiple layers of rerendering of the scene once complexity needs become high enough.

But obviously theres no thing that you technically could not do with rasterisation because its essentially just a different way of doing the same thing. Its just that some things are not feasible that way.

• This reminds me of when a student asked why do you need vector math? Tell me one thing you cannot do without them. Well obviously you can do most things without vectors but why would you want to subject yourself to that? Having a more compact notation that helps you think is a good thing. Jun 12, 2021 at 9:59
• This is a little orthogonal but: In the push for more detail, with "better" results the average triangle size gets smaller. As that happens the price to rasterize all those triangles goes up while the price of casting rays for the same effect remains largely constant eventually there is a tipping point where performance for both effects is either very close, or ray casting actually starts to be cheaper (especially with global lighting techniques). When given two rendering techniques with roughly the same performance folks will almost always choose the one that is easiest to implement. Jun 12, 2021 at 13:18

Rasterization is based on the idea of projecting various primitives (e.g. triangles, line segments, points, quads, maybe even some curved surfaces like bezier patches) on the screen and then rasterizing those in 2D (the rasterization is the discretisation step of the otherwise continuous primitive). For a certain class of primitives (e.g. triangles) a perspective or an orthographic projection is cheap to compute, as is the subsequent rasterization step (additionally there is specialized hardware for that in GPUs), which explains the popularity of rasterization in the early days of real-time computer graphics. Note that ray-tracing is quite old too however, as Appel's paper dates back to 1968, albeit the idea can be traced to long before that.

This projection and rasterization step already impose some constraints such as the admissible camera model (although with much effort those may be relaxed somewhat). In contrast in ray-tracing one can pick an arbitrary camera model, with potentially a curved film and a whole sequence of lenses (a camera may even be a sphere or a mesh). Also the aperture can be modeled trivially in ray-tracing, achieving effects such as depth of field with ease, which is not the case for rasterization (even though some hacks may be employed). Furthermore, ray-tracing can work rather naturally with acceleration structures such as bvh and kd-trees, while this is not as natural for rasterization (albeit feasible in some sense).

Additionally, the class of primitives that can be intersected with a ray is much larger than the class of primitives that can be projected easily on the film (e.g. fractals can be ray-marched against, as well as other implicit surfaces). Furthermore, volumes can be marched through with a ray, achieving realistic participating media effects, which is much harder to do with rasterization. Following this, refraction and transparency are trivial in ray-tracing, while quite problematic (and even expensive) with rasterization.

The restrictions listed above are by no means complete, and those were only concerning what one would consider primary rays/direct visibility effects (somewhat arguable what lens systems, volumetric, transparency, and refraction effects should be classified as however).

When we get to lighting, the restrictions are even more severe, as most of real-time rasterized computer graphics rely on what's known as the Utah approximation (see Section 3). The Utah approximation considers only direct illumination, and ignores indirect illumination (global illumination) effects. The direct illumination considered is rather limited, in the sense that occlusion of light sources is ignored (i.e. no shadows), and the considered light sources are typically Dirac delta lights (those are non-physical) which allow avoiding integration (i.e. point light, directional light, spot light). Various hacks need to be used to put back in some approximations of the removed effects: e.g. shadow maps, ambient occlusion, ambient term, radiance probes, etc. In ray-tracing all of those effects arise rather naturally. This is not surprising, as ray-tracing is used in the modelling of geometric optics: how light is transported in the real world (albeit with some assumptions that do not hold in the real world).

In conclusion, you can see ray-tracing as a generalization of rasterized 3d graphics, where the latter is used because of efficiency considerations (although it is less efficient for some effects).

Edit:

That is to say, if the hardware was more powerful, I wouldn't see a reason for using projection-based rasterized graphics over ray-tracing. Unfortunately, hardware patents belonging to tech giants lock out potential competitors, which I assume leads to a slower rate of improvement of hardware. Even worse, tech giants have been able to get patents over basic mathematics that clearly does not "belong" to them (e.g. Image generation using low-discrepancy sequences).

Since we're comparing against ray-tracing, let's assume the argument is restricted to 3D rendering based (at least somewhat) on real physics.

A comparison of ray-tracing and rasterization at a high level:

## Ray-tracing:

The foundation is to replicate the behaviour of the real physical process of imaging/vision. Namely, the travel and interaction of light. It follows that any effects existing in real imaging (reflection, lighting, diffusion, etc.) can be created in ray-tracing by making the simulation more sophisticated.

## Rasterization:

The foundation is taking advantages of the outcomes of real imaging, under certain idealized circumstances. As a starting point in rasterization, we assume that the environment and resulting image follow certain regular constraints. Building on this assumption, we find a process which despite not being based on the real physics processes of imaging/vision, still produces "correct" results within the given constraints. It follows that any effects existing in real imaging that don't fall nicely in these constraints, will be difficult or impossible to create correctly with rasterization.

What are the constraints? (note that not everything is necessarily a hard constraint in every rasterization-based renderer, I'm just giving it as a starting point to demonstrate which problems don't play nicely).

• Any renderable thing can be defined (at least in its bounds) by a primitive shape
• For each object in 3d space (primitives), we can project that form it into the resulting image to create a region, which within, all of that object's image information lies.
• There is a calculable mapping between the points in projection of an object in an image, and the points in the object in 3d space.

(further constraints in most real systems)

• Primitives in both 3D and image space only consist of polygons, lines and points.
• 3D to screen projection is only usable in perspective matrix or orthographic matrix.
• The mapping between projected points and "real" points is based only on simple linear/bi-linear interpolation.

As a result, these kinds of effects are difficult to implement accurately:

• Non-parallel projections such as stereographic projection
• Projections which create an effective "field of view" that spans near or greater than 180 degrees.
• Non-local visual effects, where the visual information in a point of the image is not only determined by objects which project to that point.
• Lighting and shadows
• Reflections
• Non-planar surfaces (bumpy, etc.)
• Other effects resulting in non-linear path of light such as refraction.

These effects aren't necessarily impossible, and most have well known approximation methods in rasterization-based rendering. These approximations don't necessarily use "rasterization" per-se, they may just happen to be compatible with it, and usually still within considerable constraints.