Rotate a model or a view with mouse input to produce intuitive results

Question

I am writing 3D graphics software in Java using the LibGDX framework (which is a front-end for OpenGL), and am baffled by a transformation problem. How do I rotate the orientation of the camera or models being displayed by the dx and dy inputs from a mouse?

Click here to view the full source code of a concise demo in github.com

But, here is the method where I'm certain the problem and solution will be found:

@Override public boolean touchDragged( int screenX, int screenY, int pointer )
{
    lastX = screenX-lastX;
    lastY = screenY-lastY;

    // distance of mouse movement
    screenAng = (float) Math.sqrt( lastX*lastX + lastY*lastY );
    // direction vector of the AOR
    screenAOR.set( lastY/screenAng, lastX/screenAng, 0f );
    if ( touchedButton == 0 )
    {
        cubes[ selectedCube ].modelInstance.transform.rotate( screenAOR, screenAng );
    }
    else
    {
        camera.rotateAround( Vector3.Zero, screenAOR, -screenAng/5.5f );
        camera.update();
    }

    lastX = screenX;
    lastY = screenY;
    Gdx.graphics.requestRendering();
    return true;
}

EDIT: Thanks to Andrew Wilson for his comment below. I edited the camera section and it works properly now. Still working on the model section...

    {

        // transform the screen AOR to a world AOR

        // get the camera transformation matrix
        tempMat = new Matrix4( camera.view );
        tempMat.translate( camera.position );
        tempMat.inv();

        // transform the screen AOR to a world AOR
        worldAOR = transformThings( tempMat, screenAOR, new Vector3() ).nor();

        // apply the rotation of the angle about the world AOR to the camera
        camera.rotateAround( Vector3.Zero, worldAOR, screenAng/5.5f );
        camera.update();
    }

EDIT: I edited the model section and it still is giving me some grief - it looks like it's orbiting the world origin instead of rotating around the model origin. Here's what it looks like so far:

    {

        // transform the screen AOR to a model AOR

        // get inverse of the camera transformation matrix
        tempMat.set( camera.view );
        tempMat.translate( camera.position );
        tempMat.inv();

        // get inverse of the model transformation matrix
        Vector3 tempPos = cubes[ selectedCube ].position;
        cubes[ selectedCube ].modelInstance.transform.translate( tempPos.cpy().scl( -1 ) );
        tempMat2.set( cubes[ selectedCube ].modelInstance.transform );
        tempMat2.inv();

        // multiply the camera and model matrices together
        tempMat.mul( tempMat2 );

        // transform the screen AOR to a model AOR
        modelAOR = transformThings( tempMat, screenAOR, modelAOR ).nor();

        // apply the rotation of the angle about the model AOR to the model
        cubes[ selectedCube ].modelInstance.transform.rotate( modelAOR, -screenAng );
        cubes[ selectedCube ].modelInstance.transform.translate( tempPos );
    }

Here is an image that the demonstrates the problem:

I want the object to rotate in the direction it is dragged by the mouse, no matter which direction it happens to be orientated at the time. As it is now, when I first drag the mouse to the right, the object rotates to the right about the screen Y axis as expected; but then when I drag the mouse upward I want the object to rotate upward about the screen X axis, but instead it spins to the left about the screen Z axis. Think of it like a floating ball in a bowl of water - whichever way you swipe at it, it ought to rotate in that direction.

It seems to me that the mouse movement is transforming the objects directly in their local coordinate system; but instead I think I need to transform the axis of rotation itself from the Screen Coordinate System into the Object Coordinate System before applying it to the object. I just can't figure out how to do it.

I would really appreciate any insight or help to resolve this; I'm running out of hair to pull out... Thanks in advance.

Answer 1

Here's a more complete answer from my comment. The end product we want is:

Projection * ViewTranslate * Model * ViewRotate

All are 4x4 matrices. Where model is your models transformations. To implement you can just separate out the view's translate and rotate during your draw.

If you're using an abstraction that requires you to use Proj * View * Model. Or you just really don't want to ruin your own abstraction that follows that pattern. You'll need to modify model to cancel out views rotational part and put it on the right side:

Model = ModelTranslate * View^-1 * ViewTranslate * ModelRotate * ViewRotate 
      = ModelTranslate * ViewRotate^-1 * ModelRotate * ViewRotate

Here's some of my code that does the above. It does not matter if model or view is translated, objects will still rotate around their local origins:

glm::vec2 diff = (prevMousePos - mousePos) * rotSpeed;
glm::mat4 rotX = MathHelp::matrixRotateY(diff.x);
glm::mat4 rotY = MathHelp::matrixRotateX(diff.y);

// Decompose the translate and rotate part of view
glm::mat4 viewT, viewR;
Decompose(cam->view, viewT, viewR);

// Decompose the translate and rotate part of model
glm::mat4 modelT, modelR;
Decompose(exampleObj->model, modelT, modelR);

exampleObj->model = modelT * glm::inverse(viewR) * rotX * rotY * viewR * modelR;

This preserves the existing rotation and translation in exampleObj->model as the function gets called continually applying small rotations. And assumes you'll be using Proj * View * Model.

Lastly decompose just separates out the 3x3 rotation/scaling from the translational part:

void Decompose(glm::mat4 m, glm::mat4& t, glm::mat4& r)
{
    glm::mat4 I4 = glm::mat4(1.0f); // Identity
    for (unsigned int i = 0; i < 4; i++)
    {
        for (unsigned int j = 0; j < 4; j++)
        {
            if (i == 3 || j == 3)
            {
                r[i][j] = I4[i][j];
                t[i][j] = m[i][j];
            }
            else
            {
                r[i][j] = m[i][j];
                t[i][j] = I4[i][j];
            }
        }
    }
}

Answer 2

Here is the change to the code that made the whole thing work correctly:

@Override public boolean touchDragged( int screenX, int screenY, int pointer )
{
    lastX -= screenX;
    lastY -= screenY;

    // distance of mouse movement
    screenAng = Vector3.len( lastX, lastY, 0f );
    // direction vector of the AOR
    screenAOR.set( lastY/screenAng, lastX/screenAng, 0f );

    if ( touchedButton == 0 )
    {   // rotate the part

        // transform the screen AOR to a model rotation

        Matrix4 camT, camR, camRi, modT, modR;
        camT = new Matrix4();
        camR = new Matrix4();
        modT = new Matrix4();
        modR = new Matrix4();

        decompose( camera.view, camT, camR );
        camRi = camR.cpy().inv();

        decompose( cubes[ selectedCube ].modelInstance.transform, modT, modR );

        tempMat.idt()
                .mul( modT )
                .mul( camRi )
                .rotate( screenAOR, -screenAng )
                .mul( camR )
                .mul( modR );

        cubes[ selectedCube ].modelInstance.transform.set( tempMat );
    }
    else if ( touchedButton == 1 )
    {   // rotate the camera

        // transform the AOR from screen CS to camera CS

        // get the camera transformation matrix
        tempMat.set( camera.view );
        tempMat.translate( camera.position );
        tempMat.inv();

        // transform the screen AOR to a world AOR
        worldAOR = transform( tempMat, screenAOR, worldAOR ).nor();

        // apply the rotation of the angle about the world AOR to the camera
        camera.rotateAround( Vector3.Zero, worldAOR, screenAng/5.5f );
        camera.update();
    }

    lastX = screenX;
    lastY = screenY;
    Gdx.graphics.requestRendering();
    return true;
}

// --------------------------------------------------------------------- //
Vector3 transform( Matrix4 mat, Vector3 from, Vector3 to )
{
    // transform a vector according to a transformation matrix

    to.x = from.dot( mat.val[ Matrix4.M00 ], mat.val[ Matrix4.M01 ],
            mat.val[ Matrix4.M02 ] )+mat.val[ Matrix4.M03 ];
    to.y = from.dot( mat.val[ Matrix4.M10 ], mat.val[ Matrix4.M11 ],
            mat.val[ Matrix4.M12 ] )+mat.val[ Matrix4.M13 ];
    to.z = from.dot( mat.val[ Matrix4.M20 ], mat.val[ Matrix4.M21 ],
            mat.val[ Matrix4.M22 ] )+mat.val[ Matrix4.M23 ];
    return to;
}

// --------------------------------------------------------------------- //
void decompose( Matrix4 m, Matrix4 t, Matrix4 r )
{
    Matrix4 I4 = new Matrix4(); // Identity
    for ( int i = 0; i < 4; i++ )
    {
        for ( int j = 0; j < 4; j++ )
        {
            if (i == 3 || j == 3)
            {
                r.val[ i*4+j ] = I4.val[ i*4+j ];
                t.val[ i*4+j ] = m.val[ i*4+j ];
            }
            else
            {
                r.val[ i*4+j ] = m.val[ i*4+j ];
                t.val[ i*4+j ] = I4.val[ i*4+j ];
            }
        }
    }
}