# What quad will fill the screen, which can be inverse-projection-view transformed to map to the near frustum plane?

I've been experimenting with raytracing using GLSL. I had success rendering a cube which encloses the object.

Now I want to do a full-screen raytrace, and I want to construct a quad which covers the screen. This isn't working.

First, for the working version, the shaders looked something like this:

in vec3 v_position;

uniform mat4 u_proj_view;

void main ()
{
gl_Position = u_proj_view * vec4 (v_position, 1.0);
}


Fragment shader, which computes the word coordinate W of the fragment, and given the eye coordinate E solves P=E+t(W-E) for t given some surface intersection constraint on P.

vec3 W = screen_to_world (gl_FragCoord, inverse_proj_view, viewport);
vec3 EW = W - E;
// solve for t, etc etc.


The screen_to_world function I took from here

vec3 screen_to_world
(
in vec4 screen_fragment_coordinate,
in mat4 inverse_projection_view,
in vec4 viewport
)
{
vec4 S = screen_fragment_coordinate;
vec4 NDC;

NDC.xy = ((2.0 * S.xy) - (2.0 * viewport.xy))
/ (viewport.zw) - 1.0;

NDC.z = (2.0 * S.z - gl_DepthRange.near - gl_DepthRange.far) /
(gl_DepthRange.far - gl_DepthRange.near);

NDC.w = 1.0;

return vec4 (inverse_projection_view * (NDC / S.w)).xyz;
}


For the full-screen render, It would be nice to construct a quad directly in clip space, or directly in screen space. I tried using vertex data like this

Vertex full_screen_quad [4] =
{
{{-1, -1, 0.5}},
{{+1, -1, 0.5}},
{{-1, +1, 0.5}},
{{+1, +1, 0.5}}
};


And changed the vertex shader to

gl_Position = vec4 (v_position, 1.0);


which omits the projection-view transformation. I was hoping that this would still produce valid values for screen_to_world in the fragment shader, but no such luck. The screen is filled but the rays are wonky.

To investigate, I printed out the coordinates of the cube vertices when multiplied by the projection-view matrix and, when the cube was on screen, were something like

vec4(-2.704530, -12.888471, 12.998095, 13.171925)


I was expecting values in the range [0..1], so clearly I don't understand which coordinate space I'm working in, or I don't understand w-coordinates, or viewports, or something.

Are there fixed coordinates which I can directly pass to gl_Position, untransformed, which will effectively cover the near plane of the viewing frustum, exactly?

(When done correctly, passing such coordinates to the above screen_to_world will yield the world-space coordinates of the near frustum plane.)

Can I describe these independently of the projection-view matrix, and still have screen_to_world work correctly for each fragment?

Alternatively, and less ideally, can I hack some world-space coordinates, given the arguments to glm::lookAt and glm::perspective (which produce the projection*view matrix), such that they will cover the near frustum plane?

• I'm unclear on the purpose of your screen_to_world function. Yes, I know what it does, but I'm not clear on why you're using it. You're drawing a quad; the only reason you're drawing a quad is to make the fragment shader execute for each pixel. What is the point of wanting the world-space position of the quad? If you're doing ray-tracing, then what you need is the ray (position and direction) from the camera to that pixel. And that has nothing to do with the actual geometry you drew (though gl_FragCoord can be helpful in computing it). – Nicol Bolas Sep 11 '19 at 1:54
• screen_to_world is used when a raytraced object is surrounded by a hull of polygons. I render the polygons (rather than the whole screen) and then use the world-space coordinate of the fragment of the polygon to construct a ray which may or may not hit the object behind it. If I can execute a fragment shader for every fragment of the screen and obtain the world-space coordinate of that fragment without using this function, then great. I was just showing you my approach so far, and how it fails for the full-screen case. – spraff Sep 11 '19 at 12:27
• My point is that you don't need the "the world-space coordinate of the fragment of the polygon" in order to "construct a ray". You only need the ray itself, which can easily be constructed in camera-space, without ever knowing anything about the "hull of polygons". You're making everything much harder than it needs to be. – Nicol Bolas Sep 11 '19 at 13:26
• But I can only construct a ray in camera-space if I am raytracing an object which is also described in camera-space, right? And even then I still need the frament coordinate to be in camera-space, I still to transform gl_FragCoord to undo the projection. – spraff Sep 11 '19 at 13:29
• ... why wouldn't your object/scene be in camera-space, if that's the space that's trivial to compute your ray in? Alternatively, if it's that important (and you want to disregard all the warnings about direct usage of world-space), world-space is a single matrix transform away from camera space. It's still less work than your screen_to_world function. – Nicol Bolas Sep 11 '19 at 13:32