How are mipmap levels computed in Metal?

Question

My question is specifically in regards to Metal, since I don't know if the answer would change for another API.

What I believe I understand so far is this:

• A mipmapped texture has precomputed "levels of detail", where lower levels of detail are created by downsampling the original texture in some meaningful way.

• Mipmap levels are referred to in descending level of detail, where level 0 is the original texture, and higher-levels are power-of-two reductions of it.

• Most GPUs implement trilinear filtering, which picks two neighboring mipmap levels for each sample, samples from each level using bilinear filtering, and then linearly blends those samples.

What I don't quite understand is how these mipmap levels are selected. In the documentation for the Metal standard library, I see that samples can be taken, with or without specifying an instance of a lod_options type. I would assume that this argument changes how the mipmap levels are selected, and there are apparently three kinds of lod_options for 2D textures:

• bias(float value)
• level(float lod)
• gradient2d(float2 dPdx, float2 dPdy)

Unfortunately, the documentation doesn't bother explaining what any of these options do. I can guess that bias() biases some automatically chosen level of detail, but then what does the bias value mean? What scale does it operate on? Similarly, how is the lod of level() translated into discrete mipmap levels? And, operating under the assumption that gradient2d() uses the gradient of the texture coordinate, how does it use that gradient to select the mipmap level?

More importantly, if I omit the lod_options, how are the mipmap levels selected then? Does this differ depending on the type of function being executed?

And, if the default no-lod-options-specified operation of the sample() function is to do something like gradient2D() (at least in a fragment shader), does it utilize simple screen-space derivatives, or does it work directly with rasterizer and interpolated texture coordinates to calculate a precise gradient?

And finally, how consistent is any of this behavior from device to device? An old article (old as in DirectX 9) I read referred to complex device-specific mipmap selection, but I don't know if mipmap selection is better-defined on newer architectures.

Answer 1

Mip selection is pretty well standardized across devices today—with the exception of some of the nitty-gritty details of anisotropic filtering, which is still up to the individual GPU manufacturers to define (and its precise details are generally not publicly documented).

A good place to read about mip selection in detail is in the OpenGL spec, section 8.14, "Texture Minification". I'd assume it works the same way in Metal. (Apple could have changed something, considering they make both the hardware and the API...but I doubt they have.) I'll summarize it here.

The default mip selection (not using any of the lod_options modifiers) uses the screen-space gradients of the texture coordinates to pick the mip levels. Essentially, it tries to pick the mip levels that produce as close as possible to a 1:1 mapping of texels to pixels. For example, if the gradients have a length of 4 texels per pixel, it would pick mip level 2 (which is 1/4th the size of level 0 and therefore gives you 1 mipped texel per pixel).

With trilinear filtering, since you usually don't land on an exact 1:1 mapping, it picks the two nearest levels and linearly interpolates between them, so that you have a smooth transition between mip levels as the camera or objects in your scene move around.

To be mathematically precise: you take the screen-space gradients of the texture coordinates, scale them by the texture size (to get them in units of "texels per pixel"), take their lengths, take the longer of the two (X and Y) gradients, and calculate its logarithm base 2. This quantity is called $\lambda$ in the OpenGL spec; it's also commonly known as the texture LOD (level of detail). It's simply a continuous version of the integer mip level index. So, the mip levels used are the two integers nearest to $\lambda$, and the fractional part of $\lambda$ is used to blend between them. For example if $\lambda = 2.8$, the GPU will sample mip levels 2 and 3 of the texture, then blend to 80% level 3 and 20% level 2.

If anisotropic filtering is turned on, then instead of simply using the longer of the two gradients, you use the ratio of them to set the number of aniso samples. For example, if the X gradient is 4 times longer than the Y gradient, you'd use 4 aniso samples, with their positions spaced out along the X gradient. Each one would be a trilinear sample, using a $\lambda$ corresponding to 1/4th the length of the X gradient (i.e. two mip levels lower, since $\log_2(1/4) = -2$).

Now, as for the modifier options:

• bias applies an offset to $\lambda$ before using it to select mip levels. So, for example, a bias of +1 will make it do trilinear filtering using mips one level higher than it would normally. A bias of −1 will use mips one level lower, and so on.
• level completely overrides the automatic calculation of $\lambda$ from screen-space gradients, and lets you put in your own value of $\lambda$ (aka lod) directly.
• gradient2d lets you put in your own gradient vectors, which substitute for the implicit screen-space gradients of the texture coordinates. The rest of the mip selection and sampling process proceeds as normal, but with the altered gradients. This lets you customize anisotropic filtering.

In other types of shaders besides fragment shaders, there's no notion of "screen-space gradients", so the latter two operations are typically the only ones allowed—any operation that tries to use implicit gradients would give a compile error. I'm not positive that's how Metal does it, but that's what I'd expect from working with other APIs.