How to translate mouse movement to a rotation gizmo?

Question

I need to implement translation, scaling and rotation gizmos and I'm having trouble with the rotation gizmo. Basically I don't know how to translate the mouse movement to determine in which direction to rotate the object.

By rotation gizmo (widget) I mean something like the below:

For the translation and scaling gizmos, which are represented as 3 axes, I do the following: I make a vector of the difference between the previous and current mouse position in screen space. Then multiply this vector by the inverse of the view and projection matrices, which gives me a vector of the mouse movement in world space. Then I take the dot product of the vector in world space with the selected axis. This allows me determine if the mouse movement is in line with the axis or away from the axis.

But I can't do the same thing for the rotation gizmo. The rotation gizmo is represented as 3 circles, each representing a rotation around an axis. Once a circle is clicked, I need to find some way to translate the mouse movement to determine which direction to rotate the object around the selected axis. I don't know how to do this. Please help.

It is important to note that I am using color picking and not raycasting to determine where the gizmo was clicked. This means that I don't have the point at which the ray hits the gizmo in world space to start from. I mention this because I've looked at examples that use raycasting like libgizmo and I don't think I can do the same thing.

Thanks in advance for any help.

Answer 1

Ratchet Freak's answer might work. I'm not sure, I didn't spend much time on it because I didn't know how to create the quaternion from the dot and cross product as described.

What we ended up doing was:

• Project the center of the gizmo to screen space.

• Get the angle between the vectors GizmoToLastMouse and GizmoToCurrentMouse using the dot product formula: acos(Dot(GizmoToLastMouse, GizmoToCurrentMouse) / (GizmoToLastMouse.Length() * GizmoToCurrentMouse.Length()))

• Get the sign of the angle by taking the cross of the GizmoToLastMouse and GizmoToCurrentMouse and dotting it with the screen normal.

• Get another sign (times 1.0f or -1.0f) based on the orientation of the gizmo, by taking the dot product between the rotated axis and a vector from the center of the gizmo to the camera

• Apply the rotation based on the angle and the two signs to the appropriate axis

There might be a better way to do it, but this works pretty well for our needs.

Answer 2

Map the mouse position to the circle by drawing an imaginary line between the mouse and the center of the gizmo on the plane of the current circle. Do this for the start and current positions.

Then you can easily get the quaternion rotation around the center from the start to the end by doing normalize(quat(1+dot(v_1, v_2), cross(v_1, v_2))).

This works because the dot product is the cos of the angle between the vectors and the cross is the vector perpendicular to the 2 vectors (aka the rotation angle) and has length of the sin of the angle between the vectors. This means that you can create the double quaternion out of them. Then you can nlerp that double quaternion with the unit quaternion to get the rotation you need.

However with that information you can do an arcball by projecting the mouse onto the ball that is the gizmo and using the start and end vectors that way.