Rotate a model or a view with mouse input to produce intuitive results

I am writing 3D graphics software in Java using the LibGDX framework (which is a front-end for OpenGL), and am baffled by a transformation problem. How do I rotate the orientation of the camera or models being displayed by the dx and dy inputs from a mouse?

Click here to view the full source code of a concise demo in github.com

But, here is the method where I'm certain the problem and solution will be found:

@Override public boolean touchDragged( int screenX, int screenY, int pointer )
{
lastX = screenX-lastX;
lastY = screenY-lastY;

// distance of mouse movement
screenAng = (float) Math.sqrt( lastX*lastX + lastY*lastY );
// direction vector of the AOR
screenAOR.set( lastY/screenAng, lastX/screenAng, 0f );
if ( touchedButton == 0 )
{
cubes[ selectedCube ].modelInstance.transform.rotate( screenAOR, screenAng );
}
else
{
camera.rotateAround( Vector3.Zero, screenAOR, -screenAng/5.5f );
camera.update();
}

lastX = screenX;
lastY = screenY;
Gdx.graphics.requestRendering();
return true;
}


EDIT: Thanks to Andrew Wilson for his comment below. I edited the camera section and it works properly now. Still working on the model section...

    {

// transform the screen AOR to a world AOR

// get the camera transformation matrix
tempMat = new Matrix4( camera.view );
tempMat.translate( camera.position );
tempMat.inv();

// transform the screen AOR to a world AOR
worldAOR = transformThings( tempMat, screenAOR, new Vector3() ).nor();

// apply the rotation of the angle about the world AOR to the camera
camera.rotateAround( Vector3.Zero, worldAOR, screenAng/5.5f );
camera.update();
}


EDIT: I edited the model section and it still is giving me some grief - it looks like it's orbiting the world origin instead of rotating around the model origin. Here's what it looks like so far:

    {

// transform the screen AOR to a model AOR

// get inverse of the camera transformation matrix
tempMat.set( camera.view );
tempMat.translate( camera.position );
tempMat.inv();

// get inverse of the model transformation matrix
Vector3 tempPos = cubes[ selectedCube ].position;
cubes[ selectedCube ].modelInstance.transform.translate( tempPos.cpy().scl( -1 ) );
tempMat2.set( cubes[ selectedCube ].modelInstance.transform );
tempMat2.inv();

// multiply the camera and model matrices together
tempMat.mul( tempMat2 );

// transform the screen AOR to a model AOR
modelAOR = transformThings( tempMat, screenAOR, modelAOR ).nor();

// apply the rotation of the angle about the model AOR to the model
cubes[ selectedCube ].modelInstance.transform.rotate( modelAOR, -screenAng );
cubes[ selectedCube ].modelInstance.transform.translate( tempPos );
}


Here is an image that the demonstrates the problem:

I want the object to rotate in the direction it is dragged by the mouse, no matter which direction it happens to be orientated at the time. As it is now, when I first drag the mouse to the right, the object rotates to the right about the screen Y axis as expected; but then when I drag the mouse upward I want the object to rotate upward about the screen X axis, but instead it spins to the left about the screen Z axis. Think of it like a floating ball in a bowl of water - whichever way you swipe at it, it ought to rotate in that direction.

It seems to me that the mouse movement is transforming the objects directly in their local coordinate system; but instead I think I need to transform the axis of rotation itself from the Screen Coordinate System into the Object Coordinate System before applying it to the object. I just can't figure out how to do it.

I would really appreciate any insight or help to resolve this; I'm running out of hair to pull out... Thanks in advance.

• An intuitive rotation depends on the view direction. Multiply your change in transformation by the inverse view matrix. Then apply the change in transformation to the overall transformation of the object. Think of it like undoing whatever rotation you did with the view. If you have a translation in your camera you'll have to deal with that also. Dec 20, 2018 at 0:25
• Ok. Now I'm making progress. I got the camera rotation working properly: I created a temporary matrix from the camera, translated the position out of it, inverted it, then rotated the screenAOR about it, and Boom! It worked! Yahoo.
– Mike
Dec 20, 2018 at 5:54
• I added the working code above as an EDIT.
– Mike
Dec 20, 2018 at 6:10
• I'm still having trouble getting the model to rotate properly. I translate and invert the camera into a temporary matrix (as in the view rotation section), then I translate the invert the model instance (in a similar manner) into another temporary matrix, then I multiply them together and transform the screen AOR into a model AOR, then rotate the model instance, and lastly translate the model back to it's position.
– Mike
Dec 22, 2018 at 5:16
• The model seems to be rotating properly, except that it is as though it didn't get translated to the origin to be rotated, because as it rotates it is also orbiting the origin. Dang! @Andrew can you see anything I'm missing? I'll add the code in another EDIT above.
– Mike
Dec 22, 2018 at 5:58

Here's a more complete answer from my comment. The end product we want is:

Projection * ViewTranslate * Model * ViewRotate


All are 4x4 matrices. Where model is your models transformations. To implement you can just separate out the view's translate and rotate during your draw.

If you're using an abstraction that requires you to use Proj * View * Model. Or you just really don't want to ruin your own abstraction that follows that pattern. You'll need to modify model to cancel out views rotational part and put it on the right side:

Model = ModelTranslate * View^-1 * ViewTranslate * ModelRotate * ViewRotate
= ModelTranslate * ViewRotate^-1 * ModelRotate * ViewRotate


Here's some of my code that does the above. It does not matter if model or view is translated, objects will still rotate around their local origins:

glm::vec2 diff = (prevMousePos - mousePos) * rotSpeed;
glm::mat4 rotX = MathHelp::matrixRotateY(diff.x);
glm::mat4 rotY = MathHelp::matrixRotateX(diff.y);

// Decompose the translate and rotate part of view
glm::mat4 viewT, viewR;
Decompose(cam->view, viewT, viewR);

// Decompose the translate and rotate part of model
glm::mat4 modelT, modelR;
Decompose(exampleObj->model, modelT, modelR);

exampleObj->model = modelT * glm::inverse(viewR) * rotX * rotY * viewR * modelR;


This preserves the existing rotation and translation in exampleObj->model as the function gets called continually applying small rotations. And assumes you'll be using Proj * View * Model.

Lastly decompose just separates out the 3x3 rotation/scaling from the translational part:

void Decompose(glm::mat4 m, glm::mat4& t, glm::mat4& r)
{
glm::mat4 I4 = glm::mat4(1.0f); // Identity
for (unsigned int i = 0; i < 4; i++)
{
for (unsigned int j = 0; j < 4; j++)
{
if (i == 3 || j == 3)
{
r[i][j] = I4[i][j];
t[i][j] = m[i][j];
}
else
{
r[i][j] = m[i][j];
t[i][j] = I4[i][j];
}
}
}
}

• Wahoo! That did it! It took a little trial-and-error to get the Java translation correct, but it works perfectly. Thanks a million!!! I've added the working code as another answer below. it won't let me up-vote your answer, but this is it.
– Mike
Dec 30, 2018 at 7:24

Here is the change to the code that made the whole thing work correctly:

@Override public boolean touchDragged( int screenX, int screenY, int pointer )
{
lastX -= screenX;
lastY -= screenY;

// distance of mouse movement
screenAng = Vector3.len( lastX, lastY, 0f );
// direction vector of the AOR
screenAOR.set( lastY/screenAng, lastX/screenAng, 0f );

if ( touchedButton == 0 )
{   // rotate the part

// transform the screen AOR to a model rotation

Matrix4 camT, camR, camRi, modT, modR;
camT = new Matrix4();
camR = new Matrix4();
modT = new Matrix4();
modR = new Matrix4();

decompose( camera.view, camT, camR );
camRi = camR.cpy().inv();

decompose( cubes[ selectedCube ].modelInstance.transform, modT, modR );

tempMat.idt()
.mul( modT )
.mul( camRi )
.rotate( screenAOR, -screenAng )
.mul( camR )
.mul( modR );

cubes[ selectedCube ].modelInstance.transform.set( tempMat );
}
else if ( touchedButton == 1 )
{   // rotate the camera

// transform the AOR from screen CS to camera CS

// get the camera transformation matrix
tempMat.set( camera.view );
tempMat.translate( camera.position );
tempMat.inv();

// transform the screen AOR to a world AOR
worldAOR = transform( tempMat, screenAOR, worldAOR ).nor();

// apply the rotation of the angle about the world AOR to the camera
camera.rotateAround( Vector3.Zero, worldAOR, screenAng/5.5f );
camera.update();
}

lastX = screenX;
lastY = screenY;
Gdx.graphics.requestRendering();
return true;
}

// --------------------------------------------------------------------- //
Vector3 transform( Matrix4 mat, Vector3 from, Vector3 to )
{
// transform a vector according to a transformation matrix

to.x = from.dot( mat.val[ Matrix4.M00 ], mat.val[ Matrix4.M01 ],
mat.val[ Matrix4.M02 ] )+mat.val[ Matrix4.M03 ];
to.y = from.dot( mat.val[ Matrix4.M10 ], mat.val[ Matrix4.M11 ],
mat.val[ Matrix4.M12 ] )+mat.val[ Matrix4.M13 ];
to.z = from.dot( mat.val[ Matrix4.M20 ], mat.val[ Matrix4.M21 ],
mat.val[ Matrix4.M22 ] )+mat.val[ Matrix4.M23 ];
}

// --------------------------------------------------------------------- //
void decompose( Matrix4 m, Matrix4 t, Matrix4 r )
{
Matrix4 I4 = new Matrix4(); // Identity
for ( int i = 0; i < 4; i++ )
{
for ( int j = 0; j < 4; j++ )
{
if (i == 3 || j == 3)
{
r.val[ i*4+j ] = I4.val[ i*4+j ];
t.val[ i*4+j ] = m.val[ i*4+j ];
}
else
{
r.val[ i*4+j ] = m.val[ i*4+j ];
t.val[ i*4+j ] = I4.val[ i*4+j ];
}
}
}
}