Layer rendering in Vulkan

Question

I am writing a Vulkan video manipulation engine which has the following layers:

1. Foreground (2D)
2. Scene (3D)
3. Background (2D)

Note that each layer may have multiple objects, some of which may be transparent. For 2D objects (foreground and background), I want to preserve the relative between objects. Therefore, I thought about the following technique, in order to execute the FS the least amount of times possible:

1. Draw the opaque foreground objects backwards, writing 0x02 at the stencil buffer.
2. Draw the opaque scene objects backwards (sorted by average depth), writing and testing 0x01 at the stencil buffer and writing and testing at the depth buffer.
3. Draw the opaque background objects backwards, writing and testing 0x00 at the stencil buffer.
4. Draw the transparent background forwards, writing and testing 0x00 at the stencil buffer.
5. Draw the transparent scene objects forwards (sorted by average depth), writing and testing 0x01 at the stencil buffer and writing and testing at the depth buffer.
6. Draw the transparent foreground forwards, writing and testing 0x02 at the stencil buffer.

However, this technique does not work as specified, as the relative order between transparent and opaque objects at the background and foreground won't be preserved. This other technique would satisfy the condition (regardless of cherry picking OIT issues) although it is not efficient:

1. Draw all the background objects forwards
2. Draw the opaque scene objects backwards (sorted by average depth), writing and testing at the depth buffer.
3. Draw the transparent scene objects forwards (sorted by average depth), writing and testing at the depth buffer.
4. Draw all the foreground objects forwards

I would like to know if there is any technique with Vulkan that lets me implement the first algorithm without the mentioned problems. I've thought about a secondary depth buffer in which the relative order between 2D layers gets written (write and test a constant), but this leads me to a new concern:

According to some posts, if gl_FragDepth gets written from a shader (remember that I pretend to write a constant value from an uniform buffer to the depth value), fragment shader gets fully executed, which throws away the purpose of this optimization.

Answer 1

I think you may be overcomplicating things with your idea of using the stencil. In 2D graphics layers, we usually do not need to use the depth buffer to resolve depth order between 2D elements (their depth order is determined by draw order instead). Therefore, we can reserve the nearest depth value for all the foreground elements, and the farthest depth value for all background elements.

This would work as follows, assuming depth 0.0 is the far plane and depth 1.0 is the near plane.

1. Render opaque foreground elements in front-to-back order, using depth test VK_COMPARE_OP_GREATER, and arranging in the vertex shader for all elements to be placed at depth 1.0.
2. Render opaque scene objects roughly front-to-back.
3. Render opaque background elements in front-to-back order, placing them at depth 0.0.
4. Render transparent background elements back-to-front, now using depth test VK_COMPARE_OP_GREATER_OR_EQUAL, disabling depth writes, and placing them at depth 0.0.
5. Render transparent scene objects back-to-front.
6. Render transparent foreground elements back-to-front, placing them at depth 1.0.

The switch from GREATER to GREATER_OR_EQUAL is important here. Since the opaque elements are going front-to-back, you want to exclude the "equal" case, since you want later elements to be occluded by earlier elements. Then, when coming back-to-front for the transparents, you want to include the "equal" so that later elements will correctly render on top of earlier ones in the foreground/background layers.

Edit: Now that I'm thinking about this more, this may not work since transparent elements in foreground/background won't be able to be occluded by opaque elements in the same layer. This could be addressed by pre-clipping the transparent element geometry, or if this is important to support on the GPU, it could be addressed by rendering opaque elements depth-only (no fragment shader) in the first passes and then rendering all elements, both opaque and transparent, back-to-front in the later passes.