What is a stencil buffer?

Question

Wikipedia states that a stencil buffer is some arbitrary buffer a shader can use.

However, it hints that it's used for clipping, or otherwise "tightly binding" the depth and pixel buffers, slightly contradicting itself.

What does the stencil buffer really do, and how is it practically used in modern applications?

Answer 1

The stencil buffer definition by Wikipedia is indeed not great, it focuses too much on the details of modern implementations (OpenGL). I find the disambiguated version easier to understand:

A stencil is a template used to draw or paint identical letters, symbols, shapes, or patterns every time it is used. The design produced by such a template is also called a stencil.

That's what stencil meant before Computer Graphics. If you type stencil on Google Images, this is one of the first results:

As you can see, it is simply a mask or pattern that can be used to "paint" the negative of the pattern onto something.

The stencil buffer works in the exact same way. One can fill the stencil buffer with a selected pattern by doing a stencil render pass, then set the appropriate stencil function which will define how the pattern is to be interpreted on subsequent drawings, then render the final scene. Pixels that fall into the rejected areas of the stencil mask, according to the compare function, are not drawn.

---

When it comes to implementing the stencil buffer, sometimes it is indeed coupled with the depth buffer. Most graphics hardware uses a 1 byte (8 bits) stencil buffer, which is enough for most applications. Depth buffers are usually implemented using 3 bytes (24 bits), which again is normally enough for most kinds of 3D rendering. So it is only logical to pack the 8 bits of the stencil buffer with the other 24 of the depth buffer, making it possible to store each depth + stencil pixel into a 32 bit integer. That's what Wikipedia meant by:

The depth buffer and stencil buffer often share the same area in the RAM of the graphics hardware.

---

One application in which the stencil buffer used to be king was for shadow rendering, in a technique called shadow volumes, or sometimes also appropriately called stencil shadows. This was a very clever use of the buffer, but nowadays most of the rendering field seems to have shifted towards depth-based shadow maps.

Answer 2

The stencil buffer is an unsigned integer buffer, usually 8-bit nowadays, where you can fill per-pixel infomation as you wish based on the use of various operations (OpenGL Ops here for example) following a stencil test.

The stencil test is simply a per-pixel operation in which the current pixel stencil value is tested against the content of the stencil buffer. You set the condition to which the stencil test is passed ( e.g. ALWAYS, LESS etc. ).

You can decide what happens to the stencil value depending on the outcome of the test (in OpenGL using the operation I linked you above)

The stencil buffer has quite a few uses, the first ones that come to mind being:

• It can be used as a mask to selectively render certain part of the framebuffer and discard the rest. This, for example, is useful for reflections, but also lots more. When dealing with multiple resolution rendering it can be used to mark up edges that need to be rendered at an higher res. There are plenty of masking usages really.
• A very simple overdraw filter for example. You can set the stencil test to always pass and always increment the value. The value you end up with in the stencil buffer represents your overdraw. [*]

In the past it has been used also for shadowing techniques such as Shadow volume.

The poor stencil buffer is often quite underestimated, but can be a good weapon in a graphics programmer's armory.

[*] Value to be read back in a post-process using the right SRV!

EDIT: Also, worth mentioning that in Shader Model 5.1 (so D3D11.3 and D3D12) you have access to the stencil reference value via SV_StencilRef

Answer 3

A stencil buffer contains per-pixel integer data which is used to add more control over which pixels are rendered.

A stencil buffer operates similarly to a depth buffer. So similarly, that stencil data is stored in a depth buffer. While depth data determines which pixel is closest to the camera, stencil data can be used as a more general purpose per-pixel mask for saving or discarding pixels. To create the mask, use a stencil function to compare a reference stencil value -- a global value -- to the value in the stencil buffer each time a pixel is rendered.

Source: What Is a Stencil Buffer? | Microsoft Developer Network

The general idea is that you can set a value for each pixel and then set the operation (GREATER, SMALLER, EQUAL etc), when this evaluates to true the pixel is shaded and shown.

One application is realtime reflection, when rendering the reflection texture the stencil buffer is used to create a mask for where the reflection is applied. For example only apply the reflection to the glass material shaded triangles. It can also be used to create arbitrary two dimensional masks for the whole framebuffer.