# Screen to World coordinates glm::unProject

I'm having some trouble trying to figure out how to go from a coordinate on the screen, to 3D world coordinates on a plane.

My case is as seen on the picture further down. World origin is in the center with the grid representing the X-Y plane.
I want to be able to click/touch on the screen, and transform the screen coordinates to the correct 3D world coordinates on the X-Y plane.

I first tried to figure out the maths myself by simple geometry, but got stuck...
I then tried the function glm::unProject() after checking out this post.

The screen has dimensions mWindowWidth and mWindowHeight.
x, y are the screen coordinates where origin is in the top left corner. Positive x to the right, positive y down.

projM is the projection matrix.

Third parameter is the model matrix, which I believe in my case only needs to be an identity matrix.

I get the world coordinates X and Y (Z is already known as it is on the plane) by using these parameters in glm::unProject:

glm::vec3 posVec = glm::unProject(
glm::vec3(x, float(mWindowHeight) - y, 1.0f),
glm::mat4(1.0f),
projM,
glm::vec4(0.0f, 0.0f, float(mWindowWidth), float(mWindowHeight))
);


What happens is that if I have screen coordinates near world origin, the world coordinates seems correct. But If I use screen coordinates near the edges and corners of the screen, I get a certain offset (see image below).

Where is this offset coming from, and what am I misunderstanding?

EDIT:

As mentioned in the comments, the second parameter should be the "model-view" matrix. I tried to switch glm::mat4(1.0f) with viewM (view matrix). The effect is still the same, if not with even more offset.

I also tried to get the z buffer depth value by using: (Got it from this tutorial)

float z;
glReadPixels(int(x), int(mWindowHeight) - int(y), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &z);


This results in very small values of X and Y world coordinates, and essentially where ever I touch, the resulting coordinates are near the origin.

• Does it work better if the plane is facing the camera? I don't think you're giving it the correct Z. I don't know that particular unproject function but I suspect it should be Z from your depth buffer which is certainly not constant with the plane orientation shown. Jan 10, 2017 at 1:45
• I get a similar result with the plane facing the camera.. I'm not sure how to get that z value from the depth buffer. But a value different from 1 just seem to make it worse. Jan 10, 2017 at 5:52
• You are not passing the view matrix. In the StackOverflow link you posted, it states that you should be passing the ModelView matrix on the 'model' parameter. That makes sense, since the function cannot guess which view matrix you are using and it needs to know that to reverse all the transformations.
– Samu
Jan 10, 2017 at 10:40
• Ok, yes good point. I think I assumed since the function takes in the window dimensions it had the parameters needed for the "view". But this makes more sence, I didn't understand why the model matrix was a parameter. Will try this later, thanks. Jan 10, 2017 at 10:44
• I tried with the view matrix. See edits. Jan 10, 2017 at 19:48

You can calculate the world position of the pixel on near plane quite easily by first defining Normalized Device Coordinates (NDC) for the point and then transforming the NDC back to the world space. You can calculate NDC for your point as follows: $$v=[2*(x+0.5)/width-1, 1-2*(y+0.5)/height, 0, 1]$$ I'm using 0 here for the z-component assuming near-plane in NDC is 0, but it could be something else as well, so check with your projection matrix what values comes out at the near-plane. This vector must be then transformed to clip-space by multiplying by near-plane distance: $$v'=np*v$$ Now that you have clip-space coordinates, you just need to transform this back to world space with the standard clip$\rightarrow$world transformation matrix: $$v''=M*v'$$
• If you transform a point $[0, 0, np]$ from camera space to NDC, what's the value you get for z-coordinate. And to your second question, no, they are in NDC. You have to multiply by w (i.e. near-plane distance in your case) to get them into clip-space Jan 11, 2017 at 15:14