How to perform vertex selection using unproject (Qt/openGL)

Question

I am struggling trying to implement vertex selection by mouse clicking. I load in some set of vertices and display them onto the screen. Then I do the following to try and retrieve a line that passes through the selected point in world coordinates and through the eye position.

  setFocus();
  float xRatio, yRatio, xScene, yScene;
  xRatio = (float)event->x() / width();
  yRatio = (float)event->y() / height();

  xScene = (1-xRatio)*-1 + xRatio*1;
  yScene = yRatio*-1 + (1-yRatio)*1;
  // mouse click now transformed to [-1,1]^2

  // Import the viewPort
      GLint *params = new GLint(4);
      glGetIntegerv(GL_VIEWPORT, params);
      QRect vp = QRect(*params, *(params + 1), *(params + 2), *(params + 3));     


      QVector3D dir = QVector3D(xScene, yScene, 0.2); // distance to near pane = 0.2
      QVector3D origin = QVector3D(0.0, 0.0, 0.0);

  // Now I have the origin and point in near clipping pane of the line (I think)
  // Unproject them to world coordinates.
      dir = dir.unproject(modelViewMatrix, projectionMatrix, vp);
      origin = origin.unproject(modelViewMatrix, projectionMatrix, vp);

  float minDist = 1000.0;
  int minIndex = 0;
  float dist;

  for (int i = 0; i < vertexCoords.size(); ++i){
      dist = vertexCoords[i].distanceToLine(origin, dir);
      if (dist < minDist){
          minDist = dist;
          minIndex = i;
      }
  }

I keep getting wrong result with this method. It will almost always select the same point and report a large distance of about 20 to that point. So something is going terribly wrong. Can anyone spot what is going wrong or what I am missing?

Additionally, here is how the view and projection matrices look:

  modelViewMatrix.setToIdentity();
  modelViewMatrix.translate(QVector3D(0.0, 0.0, -3.0));
  modelViewMatrix.scale(QVector3D(1.0, 1.0, 1.0));
  modelViewMatrix.rotate(rotX, 0.0, 1.0, 0.0);
  modelViewMatrix.rotate(rotY, 1.0, 0.0, 0.0);

  projectionMatrix.setToIdentity();
  projectionMatrix.perspective(FoV, dispRatio, 0.2, 8.0);

Sorry for the long question, I have found quite some resources online but none of them help me figure out what is going wrong. Also, if there is a better method I would gladly hear it! Thanks

Answer 1

You're basically mixing up your spaces.

If you look at the documentation for QVector3D::unproject, you see that it actually expects coordinates in window space. And this makes sense, since you explicitly query the viewport (i.e. basically the window's pixel dimensions) and give that to the function.

But with your ratio computation above you already transformed your coordinates from that space into normalized device space, i.e. the $[-1,1]$ square. So this whole computation isn't necessary at all and is already done for you by unproject. Likewise are the near and far distances from world/eye space not the same in window space, which uses the normalized $[0,1]$ depth range. If we assume that your viewport is your whole window, then all you need to do is put those pixel coordinates you clicked on into unproject (still accounting for the different Y-convention, though):

dir = QVector3D(event.x, height()-event.y, 0.);    //in window space 0 = near, 1 = far
dir = dir.unproject(modelViewMatrix, projectionMatrix, vp);

A problem is that you don't easily know the window space coordinates of the camera position, since it is in front of the near plane, but the transformation from $[0,1]$ to $[near,far]$ is non-linear. One way to overcome this would be to just unproject a point on the near plane (depth = 0) as origin and the difference to the unproject of that same point on the far plane (depth = 1) as the direction.

QVector3D near(QVector3D(event.x, height()-event.y, 0.);
QVector3D far( QVector3D(event.x, height()-event.y, 1.);
origin = near.unproject(modelViewMatrix, projectionMatrix, vp);
dir = far.unproject(modelViewMatrix, projectionMatrix, vp) - origin;

Or you compute dir just the same and don't unproject the origin at all, rather than just transforming the eye-space origin (which is conveniently $\mathbf{0}$) back into model space (assuming your modelview matrix isn't projective, which it really shouldn't be, otherwise don't forget the W-division):

origin = QVector3D(modelViewMatrix.inverted() * QVector4D(0., 0., 0., 1.));
//origin = QVector3D(modelViewMatrix.inverted().column(3));    //would work the same

If your viewport is not the same as your window dimensions however, you need to fiddle with the mouse coordinates a little more and transform them from the entire window into your viewport rect before unprojecting them. But the point is that

unproject expects window space (i.e. pixel) coordinates.