# OpenGL - Recreate position in viewspace from depth values

Current state:

I belive the reconstructed positions are correct. However, I still have the same problem as all the time. When I rotate around my single pointlight appears to be moving around also.

I finally solved this! It was annoying, but I did learn alot on the way.

The explanation

I am trying to recreate viewspace positions from only a stored depth value, and then calculate the light in viewspace.

In the first pass I store the Normals and the normalized depth in viewspace. In the second pass I try to build reconstruct the position in viewspace.

I belive something is going wrong in the reconstruction, but I can't see what, or if it something else. I did try to reconstruct the position in different ways, always with the same not good result.

The light initially looks fine, but when I start moving around the light seems to be moving around. When I go futher away, things get less lit. When rotating the light also rotates. Note: When I render out the actual depth values, it looks fine, so think the depth is fine.

What could I be doing wrong?

Belove is code for the fragment & vertex shader, first and second pass.

Thank you

First pass:

--Vertex
#version 460

layout( location = 0 ) in vec3 VertexPosition;
layout( location = 1 ) in vec3 VertexNormal;

/* Uniforms */
uniform mat4 ViewMatrix;
uniform mat4 ProjectionMatrix;
uniform mat3 NormalMatrix;
uniform float FarPlane;
uniform float NearPlane;

out vec3 vNormal;
out highp float vDepth;

void main() {
mat4 vMVP = ProjectionMatrix * ViewMatrix;
gl_Position = vMVP * vec4( VertexPosition, 1.0 ); // Screenspace

vNormal = normalize(NormalMatrix  * normalize(VertexNormal));
vec3 vPos = (ViewMatrix * vec4( VertexPosition, 1.0 )).xyz;

// Save normalized depth [0, 1]
vDepth = -(vPos.z - NearPlane) / ( FarPlane - NearPlane);
}

--Fragment
#version 460

in vec3 vNormal;
in highp float vDepth;

layout (location = 0) out highp vec4 NormalDepth;

void main() {
NormalDepth = vec4(normalize(vNormal), vDepth);
}



Second pass (off rendering to screen):

--Vertex
#version 460

layout( location = 0 ) in vec3 VertexPosition;

out vec2 vTexCoord;
out vec2 vDeviceCoords;

void main() {
vDeviceCoords = 2.0 * ( VertexPosition.xy - 0.5 );// [-1, 1]
vTexCoord = VertexPosition.xy;//  [0, 1] // Just 2 triangles for the screen

gl_Position = vec4(vDeviceCoords, 1.0, 1.0 ); // [-1, 1]
}

--Fragment
#version 460

/* Uniforms */
uniform float NearPlane;
uniform float FarPlane;
uniform mat4 ViewMatrix;

in vec2 vTexCoord;
in vec2 vDeviceCoords;

layout( location = 0 ) out vec4 FragColor;
layout( binding = 0 ) uniform highp sampler2D NormalDepthTex;

vec3 diffuseValue( vec3 normal, vec3 position)
{
// Light position. Here now for testing purposes
// Stationary point light, transformed to viewspace
vec3 vLightPos = (ViewMatrix * vec4( -10.0, 0.0, 2.0, 1.0 )).xyz;

normal = normalize(normal);
const vec3 lightDir = normalize(position - vLightPos);
const float diffuseCos = max( dot( normal, -lightDir ), 0.0 );

return vec3(diffuseCos);
}

void main() {
const vec4 normalDepth = texture2D( NormalDepthTex, vTexCoord );

// Normalize depth to [0, FarPlane]
const float depth = normalDepth.w * (FarPlane - NearPlane);

// Unproject
vec3 position;
position = vec3(vDeviceCoords.x, vDeviceCoords.y, NearPlane);
position.xy *= -depth / NearPlane;
position.z = depth;

// View space
vec3 diffuseValue = diffuseValue(normalDepth.xyz, position);

FragColor = vec4( diffuseValue.x, diffuseValue.y, diffuseValue.z, 1.0 );
}



Edit 1

for the normals, they looks correct, always pointing toward the camera. (also I can add, if I do the light in the first pass (not reconstructed depth), then the light is 100% fine). If we look at the light, initially it looks good. I positioned the light so It should shade as seen. But it changes and becomes wrong when I rote the camera. Also the intensity changes when I go further/closer with the camera. If we slightly rotate left for example, then the light follows (I want it in position!): The viewspace depth [0, 1] EDIT 2:

I have now reconstructed the depth in three different ways, all with the same result. I am starting to think There is something else I am doing wrong? Do I calculate the light wrong or something?

The three ways I have rebuilt (or tryed to) are:

Option 1:

    depth  = normalDepth.w; // Should be [0, 1]

// Create ray to FarPlane
// HalfVec is FarPlane * glm::tan( glm::radians( m_fov / 2.0f ) )
vec3 ray = vec3(vDeviceCoords * HalfVec, FarPlane);

// Unprojection
position.xy = ray.xy * -depth / NearPlane;
position.z = depth * FarPlane;


Option 2:

    depth  = normalDepth.w * FarPlane;

vec2 p = vDeviceCoords * NearPlane * tan(radians(Fov / 2.0));
vec2 p_ = (p / NearPlane) * -depth;

position = vec3(p_, depth);


Option 3:

    depth  = normalDepth.z;

mat4 inversePrjMat = inverse( ProjectionMatrix );
vec4 viewPosH      = inversePrjMat * vec4( vDeviceCoords.x, vDeviceCoords.y, 1.0, 1.0 );
vec3 farPlaneRay   = viewPosH.xyz / viewPosH.w;

position = farPlaneRay;
position.xy *= -depth;
position.z = depth * FarPlane;


These all options ends up the same. I have recalculated and went through the math over and over again, there is something else I am missing is my feeling.

Edit 3

was asked to upload image with the normals (R, G, B). When camera is abit rotated. Edit 4

Adding some GIF animations that shows what is going on with the light. As you can see, if I dont rotate at all. just move around, then the light is as it should.    Edit 5

I did the correction Nathan Reed pointed out. Still with no better result. This is what it looks like when I render out fract(position). The boxes I render have a size 5.0 * 5.0 * 5.0 (actually each box is made up of 5x5x5 smaller boxes) and abit more angled view at the boxes Edit 6

this is frontview, rendering vec4( fract(position.x), 0.0, 0.0, 1.0 ); This is with vec4( fract(position.y), 0.0, 0.0, 1.0 ) This is what it looks like when i render vec4( VertexPosition, 1.0 ); (the 2 triangles making up the fullscreen in the second pass) edit 7

This is what is looks like if i render the fract(vPos) from the first pass. Here I render just vPos from the first pass Edit 8

This is about how it should look like. This is when I do the light calculation directly in the first pass. It is stationary, and perfect, as it should be, in this case. Edit 9

I just rendered the constructed positions, and noticed x,y-axies are flipped need to negate x,y when multiplying with depth, since depth is negate. Also we are saving the depth in viewspace, so use it straight for the z-component. That is correct, it can be noticed by the light not fading away when moving away futher with the position.z set to depth. Also, we are multiplying xy with depth since triangle similarity. To be more precise we should divide that with the nearplane, I will do that, but I am using nearplane = 1, so wont be any practical difference right now anyways.

So I did this:

    position = vec3(vDeviceCoords.x / ProjectionMatrix, vDeviceCoords.y / ProjectionMatrix, 1.0);
position.xy *= -depth / NearPlane;
position.z  = depth;


Now the light calculated on the second pass looks very much correct, appart from rotating... The light don't even fade when going futher. As long as I dont rotate(!)

Is there something else obvious I am missing?

light, when moved, but not rotated at all: Reconstructed and Position looks the same when rendered. but when doing fract on the reconstructed position I still get this: Edit 10

Still not giving up, learning alot, but getting abit crazy at this problem. So now when I look at the reconstructed using fract(position). This below is what I get (right side is the fract part, left side is just the scene with diffuse point light moving around wrongly).

the position seems to be correct now when I look at it, as each texture on left side is one unit. and that is matching the the reconstructed position: However, as Nathan pointed out. I can look at fract(vPos) in the fist pass, that is how the reconstructed position also should look.

And still, this is how it looks like, from the first pass: That one makes less sence for me, should I not be able to see the fractions in the same way there? For me the reconstructed one makes more sence

Edit 11

New findings!

I did fract directly in the first pass, to see how it looks like. I did not think about the values getting changed when saved to texture, so I was not looking at what I though before.

Anyways, it looks exactly the same as the reconstructed values, see below! So I belive now I have the right constructed depth. But however, still having the same issues as always, light seems to rotate around when I am moving.

• can you also add an image of the normals when you make this small rotation? I think this could help understanding the problem Sep 15 '20 at 15:19
• Thank you, I added one! Sep 15 '20 at 15:28
• in the first shader (vertex) what is the normalMatrix? is it the transpose(inverse(modelViewMatrix))? If not, maybe this is your problem Sep 17 '20 at 14:39

You're not correctly inverting the mapping you used to store the depth value. When storing it you use:

vDepth = -(vPos.z - NearPlane) / ( FarPlane - NearPlane);


Then you reconstruct it as:

const float depth = normalDepth.w * (FarPlane - NearPlane);


but this misses both the offset by NearPlane, and the negative sign to flip the Z axis (since it points out of the screen in right-handed view space, but you want to store positive values). You're not getting back the same view space you started from—the space that your normals and light are in.

The correct inversion should be:

const float depth = -(normalDepth.w * (FarPlane - NearPlane) + NearPlane);


The depth values will be negative; this is correct since as mentioned Z points out of the screen in right-handed view space, so the Z values of everything you can see will be below zero. You'll have to tweak some other bits of your code to match—for instance, get rid of the negative sign when you calculate the position xy, and move the light to a negative Z value. It's this attempting to work in a space where depths are positive, while using normals in a space where depth is negative, that is most likely causing the lighting issues when rotating.

Your unprojection looks like it's missing the fact that horizontal and vertical FOVs are not the same. You need the aspect ratio in there (same way as used to calculate the projection matrix). In fact, you can just use the scale factors from the projection matrix.

vec3 position = vec3(vDeviceCoords.x / ProjectionMatrix, vDeviceCoords.y / ProjectionMatrix, 1.0);
position *= depth;


You could also precalculate those 1/ProjectionMatrix values in a separate uniform value if you liked.

Finally, I'll just note that you don't need to store a depth value explicitly in a channel of a framebuffer somewhere. It's possible to bind the actual depth buffer as a texture and read that. It's stored using a nonlinear mapping so it takes a bit more code to recover a linear depth value from it, but this ends up being the best way as you get the full precision of the hardware depth buffer, and it doesn't consume a framebuffer channel that you could use for something else. For more information, see MJP's Position from Depth 3 post.

• Thank you for your reply Nathan! I agree I was off with the nearplane (Note: Justing nearplane = 1). I think the depth value was stored with correct sign? "vDepth = -(vPos.z - NearPlane) / ( FarPlane - NearPlane);". You maybe missed the first minus sign there? (anyways did try exactly what you said). Anyways, I been trying what you said now, I get same result :(! I think only part of what you wrote that I did not get was "and move the light to a negative Z value" What do you mean by that? It is a stationary light, I did put it in a place where I want it? Sep 15 '20 at 17:47
• I been studying the method posted in the blog "MJP's Position from Depth 3 post.". I try to do like him, reconstruct the viewspace position from the stored viewspace depthvalue. I cant see what the difference is. But he do not calculate any light in viewspace, dunno if that is something I am failing on somehow. Or maybe I am just crazy, certently starting to feel crazy.... Sep 15 '20 at 17:50
• The depth value was stored with the correct sign, but then reconstructed with the wrong sign, is what I'm saying. The original sign is not getting restored by the code you posted. Sep 15 '20 at 17:52
• Re: "Move the light to a negative Z-value" never mind that, I missed that the light position is in world space not view space. In any case, after transforming by the ViewMatrix it should end up with a negative Z value since it's in front of the camera. Sep 15 '20 at 17:55
• Okok. Hmm does not help so far =/. Light still rotating aound Sep 15 '20 at 17:59

Ok. So Finally, I have solved the problem. This problem I been working on for a couple of weeks. Had to go through several books in math, gamedev etc. Went through all my code, to see if I had something I did missunderstand.

Finally I found the problem! I was trying to store normals in the texture, with values between [-1, 1], I did not think about the values stored in the texture has to be [0, 1].

So I normalized them to [0, 1]. And then in the second pass I the normalize them back to [-1, 1], and finally, after many long nights of search, the problem has been found!

Hope this will help others also! And thanks everyone who had a look!

SOLVED!

• sometimes these very tiny things cost the most time to find... But nice, that you found it. Sep 17 '20 at 15:08
• They really do! But I learned alot from having this problem. I was forced to read up on alot of stuff on the way Sep 17 '20 at 15:10