Computer Graphics Stack Exchange is a question and answer site for computer graphics researchers and programmers. It only takes a minute to sign up.
Sign up to join this community
What are Spherical Harmonics & Light Probes? How useful are they in computer graphics? What exactly do they do? I've heard the word spherical harmonics & Light Probes everywhere, from siggraph presentations to blog posts.
Recently Matt Pettineo posted a 6 part blog series on them but I still don't understand what they are.
Is it another way to improve ambient lighting?
Basics of Spherical Harmonics
Spherical Harmonics is a way to represent a 2D function on a surface of a sphere. Instead of spatial domain (like cubemap), SH is defined in frequency domain with some interesting properties and operations relevant to lighting that can be performed efficiently. With increasing "order" of SH you can represent higher frequencies (details) of functions as illustrated in the image below ($l$ is the SH order). By scaling and summing the below "basis functions" you can represent any kind of 2D function on the sphere up to the frequency defined by the functions. The basis functions are defined with "associated Legendre polynomials", but usually you don't need to derive these yourself but can use existing derivations for real spherical harmonics.
One such operation that can be performed efficiently in SH is called "convolution", which means integrating the product of two spherical 2D functions over a sphere. This is a common operation in lighting calculations, e.g. one of the function could be your incident lighting and one function the BRDF. When represented as SH, this operation is simply the dot product of two SH coefficient vectors.
Another interesting operation is the ability to do efficient low-pass filtering. Because SH is represented in frequency domain, this is simply the matter of scaling or zeroing out some of the SH coefficients. Some other operations on the other hand can be difficult to perform in SH compared to spatial domain, e.g. if you want to rotate function represented as SH, it can get quite expensive for higher order SH. So it really depends on the problem if it's suitable to be performed in SH or not.
SH is generally used to represent only low-frequency functions (i.e. smoothly changing functions) because higher frequencies requires increasing the amount of storage (SH coefficients) and processing. This is why you don't see SH being used for example to replace specular reflections on shiny surfaces. There are also Zonal Spherical Harmonics, which is can be used to reduce storage & computation for 2D functions that are rotationally symmetric about z-axis, by just storing diagonal elements of the SH coefficient matrix. Also Hemispherical Harmonics can be used if you need to only deal with hemispherical functions (also common in lighting) with the advantage of being able to represent similar frequencies as SH with less coefficients.
One operation you also need to perform is "SH projection" to transform spatial domain data to SH. You can do this operation by performing convolution of spatial domain data with SH basis functions. An interesting property of SH is that unlike spatial domain representations SH doesn't suffer from aliasing, so you don't have aliasing artifacts even when projecting to very low-order SH.
Light Probes
Now that you understand the basic operations and properties of SH, we can think how to apply them to GI. A light probe records how much light is coming from every direction to the point where the probe is located. This is a 2D function on a sphere and can be represented as SH (or 3 SH function for red, green and blue). Depending on the amount of lighting details we want to encode into the probes we can choose the SH order.
For light probes that are used only for Lambertian diffuse lighting, quite low-order SH is sufficient since the convolution is performed with cosine-lobe, which can be represented with order-2 SH (9 coefficients). The SH coefficients for light probes can be generated simply by rendering a cubemap at the point of the probe and then projecting it to SH.
When rendering geometry, few nearest light probes are taken and their results are interpolated to get the incident lighting function in a point in space. This can be done for example by interpolating the SH coefficients of nearby probes directly and then performing the convolution with pixel normal oriented cosine lobe in SH domain.
If you know what a Fourier transform is, you already almost know what spherical harmonics are: they're just a Fourier transform but on a spherical instead of a linear basis. That is, while a Fourier transform is a different way of representing a function $f(x)$, spherical harmonics are the analogous thing for polar functions $f(\theta, \phi)$.
If you don't know what a Fourier transform is, you kinda need to know before you can understand spherical harmonics. The Fourier transform lets you represent a signal as a series of sine and cosine waves, each with twice the frequency of the last. That is, you can represent the signal as its average, plus a sine wave whose wavelength is the same as the length of the signal, plus a sine wave twice that wavelength, and so on. Because the Fourier transform fixes you to these particular wavelengths, you only need to record the amplitude of each one.
We commonly use Fourier transforms to represent images, which are just 2D digital signals. It's useful because you can throw away some of the sine waves (or reduce the precision with which you store their amplitude) without significantly changing what the image looks like to human eyes. OTOH, throwing away pixels changes the look of the image a lot.
In a sampled signal like an image, if you use the same number of sine waves as there were samples (pixels) in the original image, you can reconstruct the image exactly, so once you start throwing away any frequencies, you're making the image take less storage.
Spherical harmonics are just like Fourier transforms, but instead of sine waves, they use a spherical function, so instead of linear functions (such as images), they can represent functions defined on the sphere (such as environment maps).
Just like how a standard image records all the light reaching a certain point through the image plane, a light probe records all the light reaching a certain point from all directions. They first came out of movie effects. If you want to add a computer-generated object to a real-world scene, you need to be able to light the synthetic object with the real-world lighting. To do that, you need to know what light reaches the point in the scene where the synthetic object will be. (N.B. Although I say "lighting", you're recording an image of all the light, so it can be used for reflections as well.)
Because you can't have a camera with a spherical lens that records all the light reaching a single point from all directions, you record this by taking normal photographs of a spherical mirror, and then reprojecting the images onto a sphere.
Outside of movie effects, it's more common to use light probes generated from an artificial scene. Imagine you have some expensive algorithm to compute global illumination (GI) in a scene, and you also have some smaller objects moving around in this scene (such as a game level with players in it). You can't run the whole GI algorithm every time any object moves, so you run it once with the static scene, and save light probes taken at various points in the level. Then you can get a good approximation to the GI by lighting the player with whichever light probe they're closest to.
Generally you want to filter out sharp edges in your global illumination anyway, so you want a way to represent them that's compact and easily lets you throw away high frequencies. That's what spherical harmonics are really good at! That's why you'll hear these two terms used together a lot.
You compute light probes with your expensive GI algorithm - typically in the level-design tool, or maybe once per second (instead of once per frame) if you want to include your dynamic objects to. You store those cheaply with spherical harmonics - 16 floats is enough for pretty high quality lighting, but not reflections. Then for each dynamic object you want to light, you pick the nearest light probe (or linearly interpolate several together) and use it as a uniform or constant input to your shader. It's also commonplace to use spherical harmonics to represent ambient occlusion data, and it's very cheap to convolve that with the light probe, though there's some complexity around rotating spherical harmonic functions.
Spherical harmonics
Let's say you have some data in an array but you want to represent that data with a fewer number of bytes.
One way to do that could be to express the data as a function instead of the raw values.
You could represent it as a linear function: $y=ax+b$
Then instead of storing your array of values you could store just $a$ and $b$.
Problem is, a linear equation is probably a poor approximation to your data.
You could try a quadratic instead: $y=ax^2+bx+c$
Now instead of storing $a$ and $b$, you store $a$, $b$ and $c$.
We've increased the memory storage and also the computational complexity of storing and retrieving your data, compared to the linear equation, but it is a better approximation to your array of data. We could also take it up to a cubic function or higher. Increasing the order increases the storage, computation and accuracy.
Spherical harmonics is a way of making a function that is defined on a sphere instead of an $f(x)$ like I was talking about above.
Just like in the examples above, you can use a lower order spherical harmonics function to make something with lower storage, and lower computation to calculate data, but also lower accuracy.
On the other hand, you can increase the order and get a better approximation to your original data, but at the cost of more storage space required, and more computation needed to calculate a point of data.
In the extreme, you can use as many spherical harmonic terms as you had samples in your array, and then you can reconstruct your original array exactly, but you spend a lot of computation doing that, and you're using just as much storage space as you started with.
Because of this, in practice, spherical harmonics functions don't offer much benefit if you need to represent fine details - like a sharp reflection on a sphere - but they can be cheap for data that doesn't have fine details (data that doesn't have much high frequency content). They're also useful for doing frequency-domain calculations like spectral analysis or convolution.
One such data they are good at storing is "irradiance" which is the amount of light hitting a point from other directions. It tends to look a bit blurry, which means it has only low frequency content, and is a good candidate for being stored in a spherical harmonics function.
I'll leave the light probes explanation to another person :p
