# What are Spherical Harmonics & Light Probes?

What are Spherical Harmonics & Light Probes? How useful are they in computer graphics? What exactly do they do? I've heard the word spherical harmonics & Light Probes everywhere, from siggraph presentations to blog posts.

Recently Matt Pettineo posted a 6 part blog series on them but I still don't understand what they are.

Is it another way to improve ambient lighting?

Basics of Spherical Harmonics

Spherical Harmonics is a way to represent a 2D function on a surface of a sphere. Instead of spatial domain (like cubemap), SH is defined in frequency domain with some interesting properties and operations relevant to lighting that can be performed efficiently. With increasing "order" of SH you can represent higher frequencies (details) of functions as illustrated in the image below ($l$ is the SH order). By scaling and summing the below "basis functions" you can represent any kind of 2D function on the sphere up to the frequency defined by the functions. The basis functions are defined with "associated Legendre polynomials", but usually you don't need to derive these yourself but can use existing derivations for real spherical harmonics.

One such operation that can be performed efficiently in SH is called "convolution", which means integrating the product of two spherical 2D functions over a sphere. This is a common operation in lighting calculations, e.g. one of the function could be your incident lighting and one function the BRDF. When represented as SH, this operation is simply the dot product of two SH coefficient vectors.

Another interesting operation is the ability to do efficient low-pass filtering. Because SH is represented in frequency domain, this is simply the matter of scaling or zeroing out some of the SH coefficients. Some other operations on the other hand can be difficult to perform in SH compared to spatial domain, e.g. if you want to rotate function represented as SH, it can get quite expensive for higher order SH. So it really depends on the problem if it's suitable to be performed in SH or not.

SH is generally used to represent only low-frequency functions (i.e. smoothly changing functions) because higher frequencies requires increasing the amount of storage (SH coefficients) and processing. This is why you don't see SH being used for example to replace specular reflections on shiny surfaces. There are also Zonal Spherical Harmonics, which is can be used to reduce storage & computation for 2D functions that are rotationally symmetric about z-axis, by just storing diagonal elements of the SH coefficient matrix. Also Hemispherical Harmonics can be used if you need to only deal with hemispherical functions (also common in lighting) with the advantage of being able to represent similar frequencies as SH with less coefficients.

One operation you also need to perform is "SH projection" to transform spatial domain data to SH. You can do this operation by performing convolution of spatial domain data with SH basis functions. An interesting property of SH is that unlike spatial domain representations SH doesn't suffer from aliasing, so you don't have aliasing artifacts even when projecting to very low-order SH.

Light Probes

Now that you understand the basic operations and properties of SH, we can think how to apply them to GI. A light probe records how much light is coming from every direction to the point where the probe is located. This is a 2D function on a sphere and can be represented as SH (or 3 SH function for red, green and blue). Depending on the amount of lighting details we want to encode into the probes we can choose the SH order.

For light probes that are used only for Lambertian diffuse lighting, quite low-order SH is sufficient since the convolution is performed with cosine-lobe, which can be represented with order-2 SH (9 coefficients). The SH coefficients for light probes can be generated simply by rendering a cubemap at the point of the probe and then projecting it to SH.

When rendering geometry, few nearest light probes are taken and their results are interpolated to get the incident lighting function in a point in space. This can be done for example by interpolating the SH coefficients of nearby probes directly and then performing the convolution with pixel normal oriented cosine lobe in SH domain.

• What are the benefits of using this method over the simpler Image based Lighting? – Arjan Singh Oct 29 '16 at 11:45
• You can have better quality low-frequency lighting with similar storage and performance requirements than say using cubemaps. – JarkkoL Oct 29 '16 at 14:36

# Spherical harmonics

If you know what a Fourier transform is, you already almost know what spherical harmonics are: they're just a Fourier transform but on a spherical instead of a linear basis. That is, while a Fourier transform is a different way of representing a function $f(x)$, spherical harmonics are the analogous thing for polar functions $f(\theta, \phi)$.

If you don't know what a Fourier transform is, you kinda need to know before you can understand spherical harmonics. The Fourier transform lets you represent a signal as a series of sine and cosine waves, each with twice the frequency of the last. That is, you can represent the signal as its average, plus a sine wave whose wavelength is the same as the length of the signal, plus a sine wave twice that wavelength, and so on. Because the Fourier transform fixes you to these particular wavelengths, you only need to record the amplitude of each one.

We commonly use Fourier transforms to represent images, which are just 2D digital signals. It's useful because you can throw away some of the sine waves (or reduce the precision with which you store their amplitude) without significantly changing what the image looks like to human eyes. OTOH, throwing away pixels changes the look of the image a lot.

In a sampled signal like an image, if you use the same number of sine waves as there were samples (pixels) in the original image, you can reconstruct the image exactly, so once you start throwing away any frequencies, you're making the image take less storage.

Spherical harmonics are just like Fourier transforms, but instead of sine waves, they use a spherical function, so instead of linear functions (such as images), they can represent functions defined on the sphere (such as environment maps).

# Light probes

Just like how a standard image records all the light reaching a certain point through the image plane, a light probe records all the light reaching a certain point from all directions. They first came out of movie effects. If you want to add a computer-generated object to a real-world scene, you need to be able to light the synthetic object with the real-world lighting. To do that, you need to know what light reaches the point in the scene where the synthetic object will be. (N.B. Although I say "lighting", you're recording an image of all the light, so it can be used for reflections as well.)

Because you can't have a camera with a spherical lens that records all the light reaching a single point from all directions, you record this by taking normal photographs of a spherical mirror, and then reprojecting the images onto a sphere.

Outside of movie effects, it's more common to use light probes generated from an artificial scene. Imagine you have some expensive algorithm to compute global illumination (GI) in a scene, and you also have some smaller objects moving around in this scene (such as a game level with players in it). You can't run the whole GI algorithm every time any object moves, so you run it once with the static scene, and save light probes taken at various points in the level. Then you can get a good approximation to the GI by lighting the player with whichever light probe they're closest to.

# Using them together

Generally you want to filter out sharp edges in your global illumination anyway, so you want a way to represent them that's compact and easily lets you throw away high frequencies. That's what spherical harmonics are really good at! That's why you'll hear these two terms used together a lot.

You compute light probes with your expensive GI algorithm - typically in the level-design tool, or maybe once per second (instead of once per frame) if you want to include your dynamic objects to. You store those cheaply with spherical harmonics - 16 floats is enough for pretty high quality lighting, but not reflections. Then for each dynamic object you want to light, you pick the nearest light probe (or linearly interpolate several together) and use it as a uniform or constant input to your shader. It's also commonplace to use spherical harmonics to represent ambient occlusion data, and it's very cheap to convolve that with the light probe, though there's some complexity around rotating spherical harmonic functions.

• Great answer this really clarified everything! So basically Light Probes are a easy way to calculate the lighting on moving characters which prevent us from re calculating the GI for the entire scene. Spherical Harmonics on the other hand are used to filter out the high frequencies. (Correct me if I'm wrong, I'm just trying to see if I have the right understanding). – Arjan Singh Oct 23 '16 at 16:19
• "if you use the same number of sine waves as there were samples (pixels) in the original image, you can reconstruct the image exactly", actually not true. For example square wave requires infinite number of frequencies for exact representation – JarkkoL Oct 23 '16 at 19:50
• @JarkkoL Sure, a true square wave does. But if you've discretized it by sampling, then you only need the same number of frequencies to make the error less than the sampling error. It's a handy outcome of Nyquist's theorem (that the highest frequency present in the sampled signal is half the sample rate). – Dan Hulme Oct 23 '16 at 23:06
• Ah yes, that's true of course. I suppose you need half the frequency but complex (vs real) frequency domain results. Or use DCT & real domain with twice the frequency of DFT. – JarkkoL Oct 24 '16 at 14:28

Spherical harmonics

Let's say you have some data in an array but you want to represent that data with a fewer number of bytes.

One way to do that could be to express the data as a function instead of the raw values.

You could represent it as a linear function: $y=ax+b$

Then instead of storing your array of values you could store just $a$ and $b$.

Problem is, a linear equation is probably a poor approximation to your data.

You could try a quadratic instead: $y=ax^2+bx+c$

Now instead of storing $a$ and $b$, you store $a$, $b$ and $c$.

We've increased the memory storage and also the computational complexity of storing and retrieving your data, compared to the linear equation, but it is a better approximation to your array of data. We could also take it up to a cubic function or higher. Increasing the order increases the storage, computation and accuracy.

Spherical harmonics is a way of making a function that is defined on a sphere instead of an $f(x)$ like I was talking about above.

Just like in the examples above, you can use a lower order spherical harmonics function to make something with lower storage, and lower computation to calculate data, but also lower accuracy.

On the other hand, you can increase the order and get a better approximation to your original data, but at the cost of more storage space required, and more computation needed to calculate a point of data.

In the extreme, you can use as many spherical harmonic terms as you had samples in your array, and then you can reconstruct your original array exactly, but you spend a lot of computation doing that, and you're using just as much storage space as you started with.

Because of this, in practice, spherical harmonics functions don't offer much benefit if you need to represent fine details - like a sharp reflection on a sphere - but they can be cheap for data that doesn't have fine details (data that doesn't have much high frequency content). They're also useful for doing frequency-domain calculations like spectral analysis or convolution.

One such data they are good at storing is "irradiance" which is the amount of light hitting a point from other directions. It tends to look a bit blurry, which means it has only low frequency content, and is a good candidate for being stored in a spherical harmonics function.

I'll leave the light probes explanation to another person :p

• Why the down vote? :P – Alan Wolfe Oct 23 '16 at 15:22
• It's not true that spherical harmonics are not good at representing fine details. Like the Fourier transform, they can reconstruct the original signal exactly if you keep all the frequencies. It's just that they make it easy to save space by throwing away high frequencies if you don't need them. – Dan Hulme Oct 23 '16 at 15:32
• And sorry, I thought after casting the vote it was harsh to downvote for a single dubious claim in an otherwise helpful answer, but my vote was locked in by the time I had second thoughts. – Dan Hulme Oct 23 '16 at 15:33
• Ah. Yeah it's possible, in the same way that it's possible to represent any data set with a polynomial, but in practice, both are bad choices for needing close fits to many data points. With a polynomial you need N terms of an N order function to exactly fit N data points for instance, which makes it a worse choice than just an array since it's calculation, not lookup, to get a data point out. Similarly, in practical terms, spherical harmonics are a bad choice for spherical data with high frequency content that you want to preserve. It's not a good choice in those situations. – Alan Wolfe Oct 23 '16 at 15:52
• That's true if you want to be able to reconstruct individual samples, but not all uses of Fourier require that - similarly for not all uses of SH. If you're going to do a convolution, it's much cheaper to do that in the frequency domain before transforming back to samples. Would you mind me proposing an edit to make this clearer in your answer, after I've finished my own answer? – Dan Hulme Oct 23 '16 at 15:59