Supporting multiple camera types in a deferred renderer without specializing the shaders or in the shaders

Question

Perspective Camera

A (row-major) perspective transformation matrix has the following format:

$$\begin{align} \mathrm{T} &= \begin{bmatrix} \mathrm{T}_{00} &0 &0 &0 \\ 0 &\mathrm{T}_{11} &0 &0 \\ 0 &0 &\mathrm{T}_{22} &1 \\ 0 &0 &\mathrm{T}_{32} &0 \end{bmatrix}=\begin{bmatrix} 1/x &0 &0 &0 \\ 0 &1/y &0 &0 \\ 0 &0 &-w &1 \\ 0 &0 &z &0\end{bmatrix}. \end{align}$$

This transformation matrix is used to transform (homogeneous) points from view to projection space, after which the homogeneous divide is applied to transform to NDC (Normalized Device Coordinate) space. (NDC space is technically a 3D space, but for ease of notation, I use 4D points with a $w=1$).

In a deferred renderer, we need to go the other way around while resolving the GBuffer and could use four components ($x$, $y$, $z$, $w$, see above) to transform a point from NDC to view space:

/**
 Returns the projection values from the given projection matrix to construct
 the view position coordinates from the NDC position coordinates.

 @return    The projection values from the given projection matrix to
            construct the view position coordinates from the NDC position
            coordinates.
 */
inline const XMVECTOR XM_CALLCONV GetViewPositionConstructionValues(
    FXMMATRIX projection_matrix) noexcept {
    //        [ 1/X  0   0  0 ]
    // p_view [  0  1/Y  0  0 ] = [p_view.x 1/X, p_view.y 1/Y, p_view.z (-W) + Z, p_view.z] = p_proj
    //        [  0   0  -W  1 ]
    //        [  0   0   Z  0 ]
    //
    // p_proj / p_proj.w        = [p_view.x/p_view.z 1/X, p_view.y/p_view.z 1/Y, -W + Z/p_view.z, 1] = p_ndc
    //
    // Construction of p_view from p_ndc and projection values
    // 1) p_ndc.z = -W + Z/p_view.z       <=> p_view.z = Z / (p_ndc.z + W)
    // 2) p_ndc.x = p_view.x/p_view.z 1/X <=> p_view.x = X * p_ndc.x * p_view.z
    // 3) p_ndc.y = p_view.y/p_view.z 1/Y <=> p_view.y = Y * p_ndc.y * p_view.z

    const F32 x = 1.0f / XMVectorGetX(projection_matrix.r[0]);
    const F32 y = 1.0f / XMVectorGetY(projection_matrix.r[1]);
    const F32 z = XMVectorGetZ(projection_matrix.r[3]);
    const F32 w = -XMVectorGetZ(projection_matrix.r[2]);

    return XMVectorSet(x, y, z, w);
}

Orthographic Camera

An (row-major) orthographic transformation matrix has the following format:

$$\begin{align} \mathrm{T} &= \begin{bmatrix} \mathrm{T}_{00} &0 &0 &0 \\ 0 &\mathrm{T}_{11} &0 &0 \\ 0 &0 &\mathrm{T}_{22} &0 \\ 0 &0 &\mathrm{T}_{32} &1 \end{bmatrix}=\begin{bmatrix} 1/x &0 &0 &0 \\ 0 &1/y &0 &0 \\ 0 &0 &1/z &0 \\ 0 &0 &-w &1\end{bmatrix}. \end{align}$$

This transformation matrix is used to transform (homogeneous) points from view to projection space, after which the homogeneous divide (no-op) is applied to transform to NDC (= projection) space. (So basically a non-uniform scaling followed by a translation of the z component).

We could similarly use four components ($x$, $y$, $z$, $w$, see above) to transform a point from NDC (= projection) to view space:

/**
 Returns the projection values from the given projection matrix to construct 
 the view position coordinates from the NDC position coordinates.

 @return        The projection values from the given projection matrix to 
                construct the view position coordinates from the NDC position 
                coordinates.
 */
inline const XMVECTOR XM_CALLCONV GetViewPositionConstructionValues(
    FXMMATRIX projection_matrix) noexcept {

    //        [ 1/X  0   0  0 ]
    // p_view [  0  1/Y  0  0 ] = [p_view.x 1/X, p_view.y 1/Y, p_view.z 1/Z -W, 1] = p_proj = p_ndc
    //        [  0   0  1/Z 0 ]
    //        [  0   0  -W  1 ]
    //
    // Construction of p_view from p_ndc and projection values
    // 1) p_ndc.z = p_view.z/Z -W <=> p_view.z = Z * (p_ndc.z + W)
    // 2) p_ndc.x = p_view.x/X    <=> p_view.x = X * p_ndc.x
    // 3) p_ndc.y = p_view.y/Y    <=> p_view.y = Y * p_ndc.y

    const F32 x = 1.0f / XMVectorGetX(projection_matrix.r[0]);
    const F32 y = 1.0f / XMVectorGetY(projection_matrix.r[1]);
    const F32 z = 1.0f / XMVectorGetZ(projection_matrix.r[2]);
    const F32 w = -XMVectorGetZ(projection_matrix.r[3]);

    return XMVectorSet(x, y, z, w);
}

Question

How can a deferred renderer support multiple camera types (minimally: perspective and orthographic cameras, ideally: all cameras that a forward renderer could support) without specializing the shaders or specializing in the shaders?

A straightforward solution is to specify some pre-processor directive in the shader, but this unfortunately results in a multiple of all the shader perturbations used so far for lighting in a deferred pipeline. Another solution is to pass a flag in some constant buffer to specify the used camera type, though I am not a fan of the dynamic branching introduced by this flag. Furthermore, both solutions are not as flexible as is the case for forward rendering which can handle arbitrary view-to-projection matrices.

Ideally, a (hypothetical) NDC-to-view transformation matrix should handle the transformation. Such a matrix does, however, not generally exist for perspective cameras since the view to NDC transformation reduces a 4D space to a 3D subspace.

Answer 1

Projective transformations (represented by 4×4 projection matrices) are invertible. You can go from NDC coordinates back to view space using the inverse of the projection matrix, in the same way that you go from view space to NDC. That is: take your NDC $x, y, z$ coordinates, append $w = 1$ to make a 4D vector, transform by the inverse projection matrix, then divide out $w$ to get back to 3D.

View space and NDC space are both fundamentally 3D spaces, though with a 4D homogeneous representation, so the transformation between them doesn't throw away a dimension, despite the divide by $w$. (Some information is lost, though: the sign of $w$, which corresponds to whether a point was behind or in front of the camera. However, this isn't usually a concern, since we cull or clip everything behind the camera during rasterization anyway.)

As for the shader code, it can presumably be simplified according to which components may be nonzero in the inverse projection matrix. You may only need the $x, y$ diagonal elements and the lower-right 2×2 submatrix, as long as you don't need to support off-axis projections (which have additional nonzero components in their matrix).

Answer 2

For completeness (and in addition to Nathan Reed's answer), I explicitly add the inverse projection matrices for perspective and orthographic cameras.

Perspective Camera

$$\begin{align} \mathrm{T}_{\mathrm{view \rightarrow projection}} &= \begin{bmatrix} \mathrm{T}_{00} &0 &0 &0 \\ 0 &\mathrm{T}_{11} &0 &0 \\ 0 &0 &\mathrm{T}_{22} &1 \\ 0 &0 &\mathrm{T}_{32} &0 \end{bmatrix} \! , \\ \mathrm{T}_{\mathrm{projection \rightarrow view}} &=\begin{bmatrix} 1/\mathrm{T}_{00} &0 &0 &0 \\ 0 &1/\mathrm{T}_{11} &0 &0 \\ 0 &0 &0 &1/\mathrm{T}_{32} \\ 0 &0 &1 &-\mathrm{T}_{22}/\mathrm{T}_{32}\end{bmatrix} \! . \end{align}$$

        /**
         Returns the projection-to-view matrix of this perspective camera.

         @return        The projection-to-view matrix of this perspective 
                        camera.
         */
        virtual const XMMATRIX GetProjectionToViewMatrix() const noexcept override {
            const XMMATRIX view_to_projection = GetViewToProjectionMatrix();

            const F32 m00 = 1.0f / XMVectorGetX(view_to_projection.r[0]);
            const F32 m11 = 1.0f / XMVectorGetY(view_to_projection.r[1]);
            const F32 m23 = 1.0f / XMVectorGetZ(view_to_projection.r[3]);
            const F32 m33 = -XMVectorGetZ(view_to_projection.r[2]) * m23;

            return XMMATRIX {
                 m00, 0.0f, 0.0f, 0.0f,
                0.0f,  m11, 0.0f, 0.0f,
                0.0f, 0.0f, 0.0f,  m23,
                0.0f, 0.0f, 1.0f,  m33
            };
        }

Orthographic Camera

$$\begin{align} \mathrm{T}_{\mathrm{view \rightarrow projection}} &= \begin{bmatrix} \mathrm{T}_{00} &0 &0 &0 \\ 0 &\mathrm{T}_{11} &0 &0 \\ 0 &0 &\mathrm{T}_{22} &0 \\ 0 &0 &\mathrm{T}_{32} &1 \end{bmatrix} \! , \\ \mathrm{T}_{\mathrm{projection \rightarrow view}} &=\begin{bmatrix} 1/\mathrm{T}_{00} &0 &0 &0 \\ 0 &1/\mathrm{T}_{11} &0 &0 \\ 0 &0 &1/\mathrm{T}_{22} &0 \\ 0 &0 &-\mathrm{T}_{32}/\mathrm{T}_{22} &1\end{bmatrix} \! . \end{align}$$

        /**
         Returns the projection-to-view matrix of this orthographic camera.

         @return        The projection-to-view matrix of this orthographic 
                        camera.
         */
        virtual const XMMATRIX GetProjectionToViewMatrix() const noexcept override {
            const XMMATRIX view_to_projection = GetViewToProjectionMatrix();

            const F32 m00 = 1.0f / XMVectorGetX(view_to_projection.r[0]);
            const F32 m11 = 1.0f / XMVectorGetY(view_to_projection.r[1]);
            const F32 m22 = 1.0f / XMVectorGetZ(view_to_projection.r[2]);
            const F32 m32 = -XMVectorGetZ(view_to_projection.r[3]) * m22;

            return XMMATRIX {
                 m00, 0.0f, 0.0f, 0.0f,
                0.0f,  m11, 0.0f, 0.0f,
                0.0f, 0.0f,  m22, 0.0f,
                0.0f, 0.0f,  m32, 1.0f
            };
        }