Correcting my "look-at" matrix so that it works on non-camera objects?

Question

I'm familiar with the "look-at" matrix and the goal of being able to rotate a Camera at a specific location towards a specified target.

When I tell the Camera to look at some target, there're no issues. However, when some other non-Camera object needs to "look-at" another object, such as a hypothetical Dog looking at some Food, so that the Dog "points" in its direction, it doesn't work as expected. In this case, when the look-at matrix used for the Camera is used (without modification) for the Dog, the Dog ends up looking in the opposite direction I intended it to face.

What follows is the code I've been using to build my look-at matrix:

public static Matrix4 createLookAtMatrix(
    Vector4 eyePosition,
    Vector4 targetPosition,
    Vector4 upDirection)
{
    // forward, side, up
    // 
    // f = normalize(targetPosition - eyePosition);
    // s = normalize(f × normalize(upDirection));
    // u = s × f;
    Vector4 f = targetPosition.sub(eyePosition).normalize();
    Vector4 s = f.cross(upDirection.normalize()).normalize();
    Vector4 u = s.cross(f);

    float[][] values = new float[][] {
        { s.x(),  s.y(),  s.z(), -s.dot(eyePosition) }, // row 0
        { u.x(),  u.y(),  u.z(), -u.dot(eyePosition) }, // row 1
        {-f.x(), -f.y(), -f.z(), -f.dot(eyePosition) }, // row 2
        { 0f   ,  0f   ,  0f   ,  1f                 }  // row 3
    };

    return createFrom(values);
}

What am I missing? Is there a way to build such a "look-at" matrix so that it works for both regardless of who/what the viewer is (i.e. Camera or Dog)?

I'd like to avoid depending on "special corner cases" to detect what kind of object is trying to "look-at" something else, and so on.

My system can have multiple SceneObjects at any given moment connected to the same SceneNode (SceneNodes are the objects that actually contain the transforms) and I really need a way to build a "look-at" matrix that will work regardless of which kind of object happens to get attached to the SceneNode.

---

This is a screenshot of the reference book I was reading/using when writing my code above:

Answer 1

You don't seem to get the concept right:

A look-at camera matrix is created by defining two points, the "from" point (the dog eyes for example assuming you take the position of the eyes of the dog and average them to get the point in between) and a "to" point which is where the dog looks at. So if your dog is at (3,3,3) and that you want it to look at the some food which is at (-10, 100, 20), then you need to call the lookAt() method with these 2 points and it will create a matrix that will actually orient and position the camera properly.

Now you are getting confused (probably) because most cameras in most 3D packages are positioned at the origin (by default) and looking down the z-axis (by default). You can create such camera with a lookAt() function btw that would look like that:

lookAt(Vec3f(0), Vec3f(0,0,-1));

For example. Now check what the matrix this call will produce with the matrix of the default camera in Maya. You will notice they are different. The second is an identity matrix, the first one is not.

Now for example if you create a default camera in Maya it will be position at (0,0,0) and looking down the negative z-axis. If you look at the 4x4 matrix of this camera you will get identity. And maybe what you are trying to do is to create a 4x4matrix using the lookAt function and multiply this matrix by the Maya's default camera matrix in which case, your camera will point in the wrong direction (it will be flipped along the camera's local coordinate z-axis). That's because the matrix returns by lookAt properly orient the camera (as suggested in the first paragraph) but since the default camera in Maya is already oriented along the negative-z axis, your Maya camera will look the wrong way.

So in summary:

1. If you use the lookAt method in a renderer, it will work as you expect. There's no z problem.
2. if you try to use it on a camera that is already oriented along the negative z-axis and yet has identity matrix yes indeed you will be looking the wrong way. This can be fixed easily by flipping the camera back one more time along the z-axis. Simply create another matrix that scales the object/camera by (1,1,-1).

EDIT

The OP says (see comment):

I did spot what looks like a typo: it says "(mind the direction of this vector: it is To − From not From − To)", but then you go on to write the line of code Vec3f forward = Normalize(from - to);, which contradicts what you were trying to clarify.

But a note in the reference says:

The camera is aligned along the negative z-axis. Does it mean I need to rotate the camera by 180 degrees along the y-axis or scale it up by -1 along the z-axis? Not at all. Transforming a camera is no different from transforming any other object in a scene. Keep in mind that in ray-tracing, we build the primary rays as if the camera was located in its default position. This is explained in the lesson Ray Tracing: Generating Camera Rays. This is when we actually reverse the direction of the rays. In other words, the z-coordinates of the rays' directions at this point are always negative: the camera in its default position, looks down along the negative z-axis. Those primary rays are then transformed by the camera-to-world matrix. Therefore, there is no need to account for the default orientation of the camera when the 4x4 camera-to-world matrix is built.

So if you look at the reference they use by default a right-hand coordinate system, which means x-axis points to the right, y is up and z goes out of the screen (outward, going your direction when x-axis points to the right). So they build a matrix that "follows this convention" which makes sense. Then they say in the note "yes but wait, normally the camera points down the negative z-axis, so you should be doing to - from and not from - to, and they explain that in the case of ray-tracing the fact that the camera looks down the z-axis is performed when the rays are cast (by inverting their z-coordinate).

Vec3f rayDirectionInCamSpace{ x, y, -1 }; // invert here
normalize(rayDirectionInCamSpace);
Vec3f rayDirInWorldSpace = transform(rayDirectionInCamSpace, camToWorldMat);
pixelColor = trace(rayOrigWorldSpace, rayDirInWorldSpace);

So no it's not a typo. Now the reference you are pointing out maybe assume this matrix needs to be used in some OpenGL render in which there is no possibility to invert they ray direction when they cast. So they indeed use to - from instead which is the same as multiplying the cam-to-world matrix you would get from the reference by a matrix that flips the coordinates z-axis (multiply by (1,1,1-)).

So no typo, things are quite logical in both cases. The result you get is always constrained to a lot of small details, such as the handiness of the coordinate system and how is the matrix later used down your graphics pipeline (OpenGL pipeline, ray-tracer, renderer, animation package, etc. which may follow its own conventions, etc.).

EDIT 2

Note that in the reference your provide (screenshot of book) they do target - eye but then they use the negative of that in the matrix construction. Not sure why. Then also no idea why the multiply the eye by the vectors for the translation part of the matrix. It seems to be using row-order matrices for the axis of the coordinate system but then the translation part should be at mat[3][0], mat[3][1] and mat[3][2] and not mat[0][3], mat[1][3] and mat[2][3]. Not sure what book this is but seems to be terribly wrong and confusing.

EDIT 3

Code (that seems to be the only solution):

void lookAt(const Vec3f& from, const Vec3f& to, Matrix44f &V)
{
    static const Vec3f tmp(0, 1, 0);
    Vec3f forward = normalize(from - to);
    Vec3f right = crossProduct(normalize(tmp), forward); 
    Vec3f up = crossProduct(forward, right); 

    V[0][0] = right.x;
    V[0][1] = right.y;
    V[0][2] = right.z;
    V[1][0] = up.x;
    V[1][1] = up.y;
    V[1][2] = up.z;
    V[2][0] = forward.x;
    V[2][1] = forward.y;
    V[2][2] = forward.z;

    V[3][0] = from.x;
    V[3][1] = from.y;
    V[3][2] = from.z;
}


int main(int argc, char **argv)
{
    ...
    lookAt(from, to, camToWorldMatrix);
    worldToCamMatrix = camera.camToWorldMatrix.inverse();

    Matrix44f projectionMatrix;

    Matrix44f openGLWorldToCamMatrix = worldToCamMatrix;
    Matrix44f openGLProjectionMatrix = projectionMatrix;

    while (!glfwWindowShouldClose(window)) {        
        glViewport(0, 0, width, height);

        glUniformMatrix4fv(projectionLocation, 1, GL_FALSE, (const GLfloat *)&openGLProjectionMatrix[0][0]);
        glUniformMatrix4fv(viewLocation, 1, GL_FALSE, (const GLfloat *)&openGLWorldToCamMatrix[0][0]);

        ...            
        wwapBuffers(window);

    }

    return 0;
}

Answer 2

Both the question and its answer is confusing.

First of all the matrix you have

is a column "Look at" view matrix rather than a model matrix. A correct column "Look At" model matrix would be

$$\begin{bmatrix} side_x&side_y&side_z&0\\ up_x&up_y&up_z&0\\ Forward_x&Forward_y&Forward_z&0\\ eye_x&eye_y&eye_z&1 \end{bmatrix}^T$$, which is what the other answer is saying. This means that matrix in the book is the inverse of the matrix given by me and the other answer.

First matrix will be used to transform the objects in the world space to the view space of the camera. Second matrix will be used to transform the camera from its local space to the world space such that camera is pointing at whatever the "target" vector is.

There is nothing special about the second matrix, you can use it on a dog to point to food, provided its mouth is along the +z direction in its local space.

So in summary, first matrix is a view matrix is used transform objects from world space to view and the second matrix is a regular model matrix which can be applied to any object in the scene to transform it from its local space to the world space with "eye" as its world position and +z direction looking at "target".

There is nothing wrong with either matrices just that they are used for accomplishing total different tasking.