I need to implement translation, scaling and rotation gizmos and I'm having trouble with the rotation gizmo. Basically I don't know how to translate the mouse movement to determine in which direction to rotate the object.
By rotation gizmo (widget) I mean something like the below:
For the translation and scaling gizmos, which are represented as 3 axes, I do the following: I make a vector of the difference between the previous and current mouse position in screen space. Then multiply this vector by the inverse of the view and projection matrices, which gives me a vector of the mouse movement in world space. Then I take the dot product of the vector in world space with the selected axis. This allows me determine if the mouse movement is in line with the axis or away from the axis.
But I can't do the same thing for the rotation gizmo. The rotation gizmo is represented as 3 circles, each representing a rotation around an axis. Once a circle is clicked, I need to find some way to translate the mouse movement to determine which direction to rotate the object around the selected axis. I don't know how to do this. Please help.
It is important to note that I am using color picking and not raycasting to determine where the gizmo was clicked. This means that I don't have the point at which the ray hits the gizmo in world space to start from. I mention this because I've looked at examples that use raycasting like libgizmo and I don't think I can do the same thing.
Thanks in advance for any help.