Screen to World coordinates glm::unProject

Question

I'm having some trouble trying to figure out how to go from a coordinate on the screen, to 3D world coordinates on a plane.

My case is as seen on the picture further down. World origin is in the center with the grid representing the X-Y plane.
I want to be able to click/touch on the screen, and transform the screen coordinates to the correct 3D world coordinates on the X-Y plane.

I first tried to figure out the maths myself by simple geometry, but got stuck...
I then tried the function glm::unProject() after checking out this post.

The screen has dimensions mWindowWidth and mWindowHeight.
x, y are the screen coordinates where origin is in the top left corner. Positive x to the right, positive y down.

projM is the projection matrix.

Third parameter is the model matrix, which I believe in my case only needs to be an identity matrix.

I get the world coordinates X and Y (Z is already known as it is on the plane) by using these parameters in glm::unProject:

glm::vec3 posVec = glm::unProject(
   glm::vec3(x, float(mWindowHeight) - y, 1.0f), 
   glm::mat4(1.0f), 
   projM, 
   glm::vec4(0.0f, 0.0f, float(mWindowWidth), float(mWindowHeight))
);

What happens is that if I have screen coordinates near world origin, the world coordinates seems correct. But If I use screen coordinates near the edges and corners of the screen, I get a certain offset (see image below).

Where is this offset coming from, and what am I misunderstanding?

EDIT:

As mentioned in the comments, the second parameter should be the "model-view" matrix. I tried to switch glm::mat4(1.0f) with viewM (view matrix). The effect is still the same, if not with even more offset.

I also tried to get the z buffer depth value by using: (Got it from this tutorial)

float z;
glReadPixels(int(x), int(mWindowHeight) - int(y), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &z);

This results in very small values of X and Y world coordinates, and essentially where ever I touch, the resulting coordinates are near the origin.

Answer 1

You can calculate the world position of the pixel on near plane quite easily by first defining Normalized Device Coordinates (NDC) for the point and then transforming the NDC back to the world space. You can calculate NDC for your point as follows: $$v=[2*(x+0.5)/width-1, 1-2*(y+0.5)/height, 0, 1]$$ I'm using 0 here for the z-component assuming near-plane in NDC is 0, but it could be something else as well, so check with your projection matrix what values comes out at the near-plane. This vector must be then transformed to clip-space by multiplying by near-plane distance: $$v'=np*v$$ Now that you have clip-space coordinates, you just need to transform this back to world space with the standard clip$\rightarrow$world transformation matrix: $$v''=M*v'$$