How to translate mouse movement to a rotation gizmo?

I need to implement translation, scaling and rotation gizmos and I'm having trouble with the rotation gizmo. Basically I don't know how to translate the mouse movement to determine in which direction to rotate the object.

By rotation gizmo (widget) I mean something like the below:

For the translation and scaling gizmos, which are represented as 3 axes, I do the following: I make a vector of the difference between the previous and current mouse position in screen space. Then multiply this vector by the inverse of the view and projection matrices, which gives me a vector of the mouse movement in world space. Then I take the dot product of the vector in world space with the selected axis. This allows me determine if the mouse movement is in line with the axis or away from the axis.

But I can't do the same thing for the rotation gizmo. The rotation gizmo is represented as 3 circles, each representing a rotation around an axis. Once a circle is clicked, I need to find some way to translate the mouse movement to determine which direction to rotate the object around the selected axis. I don't know how to do this. Please help.

It is important to note that I am using color picking and not raycasting to determine where the gizmo was clicked. This means that I don't have the point at which the ray hits the gizmo in world space to start from. I mention this because I've looked at examples that use raycasting like libgizmo and I don't think I can do the same thing.

Thanks in advance for any help.

• It took me a second to understand what you meant by a rotation gizmo. I added an image of one in case it's helpful (i think you can't add images until you have more reputation sadly!). If you have a more appropriate image in mind, feel free to let me know and i can change the image. – Alan Wolfe Jan 13 '17 at 18:54
• Thanks for adding the image Alan. You are right, I think it will make things clearer. – 0xfeedbacc Jan 13 '17 at 18:57
• Fwiw I think you are right. Using color based selection isn't going to work for this. – Alan Wolfe Jan 14 '17 at 15:34

Map the mouse position to the circle by drawing an imaginary line between the mouse and the center of the gizmo on the plane of the current circle. Do this for the start and current positions.

Then you can easily get the quaternion rotation around the center from the start to the end by doing normalize(quat(1+dot(v_1, v_2), cross(v_1, v_2))).

This works because the dot product is the cos of the angle between the vectors and the cross is the vector perpendicular to the 2 vectors (aka the rotation angle) and has length of the sin of the angle between the vectors. This means that you can create the double quaternion out of them. Then you can nlerp that double quaternion with the unit quaternion to get the rotation you need.

However with that information you can do an arcball by projecting the mouse onto the ball that is the gizmo and using the start and end vectors that way.

• THis answer really needs a picture – joojaa Jan 19 '17 at 7:49

Ratchet Freak's answer might work. I'm not sure, I didn't spend much time on it because I didn't know how to create the quaternion from the dot and cross product as described.

What we ended up doing was:

• Project the center of the gizmo to screen space.

• Get the angle between the vectors GizmoToLastMouse and GizmoToCurrentMouse using the dot product formula: acos(Dot(GizmoToLastMouse, GizmoToCurrentMouse) / (GizmoToLastMouse.Length() * GizmoToCurrentMouse.Length()))

• Get the sign of the angle by taking the cross of the GizmoToLastMouse and GizmoToCurrentMouse and dotting it with the screen normal.

• Get another sign (times 1.0f or -1.0f) based on the orientation of the gizmo, by taking the dot product between the rotated axis and a vector from the center of the gizmo to the camera

• Apply the rotation based on the angle and the two signs to the appropriate axis

There might be a better way to do it, but this works pretty well for our needs.

• a quaternion is built as quat(cos(angle/2), axis*sin(angle/2)) and you know that the dot product us equal to cos(angle) times the product of the lengths and the cross product is the axis times sin(angle) times the product of the lengths. Normalize it and you can easily get the quaternion of the same rotation but with double the angle. To get the proper quaternion you can do slerp(unit, quat, 0.5) which after inlining ends up as normalize(unit+quat); – ratchet freak Jan 19 '17 at 10:18