Why does opengl use 4d matrices for everything?

Question

What I managed to figure out is that the first 3*3 is used for rotation and scale and the 1*3 at the end is used for position, but what is the bottom row used for? is it only for clipping related things?

Answer 1

So,

A lot of folks would tell you that a 4x4 matrix is used so you can get a translation component rolled into your linear transformation. Meaning that with a 3x3 matrix you can only compute rotations, scales, and shears. (and some would argue that a shear is really another type of rotation) This same person might also go into great detail about affine transformations and homogeneous coordinate systems. And they would be right.

But that's not the reason.

All this might leave the astute observer thinking, well wouldn't a 3x3 matrix get the transformation out of the way then just use vector math to add our translation on at the end saving lots of special math, hardware, and precious resources?

Well this is perfectly plausible, and a little extra thinking would leave use with the idea of adding a 4th column/row to our matrix giving a 3x4 matrix. Now, a translation is suddenly perfectly plausible and would create a pretty world where you can do all the operations needed to move and deform objects all day long saving lots of math and resource. And you would be right. You can in fact do this very thing when computing your Model and View matrices. Handling vectors would require a little extra trickery but we could manage it.

So, now, on this theoretical video card, since all this has to be implemented in hardware, we have gotten or world models/meshs/blobs whatnots and whathaveyous all into place and are ready to project them from our 3D world onto our 2D screen. And we think: this too isn't a problem since a projection is really just a non-linear scaling operation that squeezes things around so the stuff we want to see falls inside our clipping volume.

A little bit of algebra later...

We have mountains of triangles all of which are ready to have their attributes computed, and interpolations...well interpolated. We then use standard interpolation, check our work very carefully and what we get on our video monitor looks like some kind of star trek space anomaly got hold of it.

This is where the problem comes up: It turns out that you can't just interpolate values directly after performing a perspective projection. Instead you must use a special form of interpolation called...wait for it...Perspective Correct Interpolation. And the solution to perspective correct interpolation leads use to the correct form of the projection matrix which has 4 rows and 4 columns. That matrix after being applied to a 4 component position vector results in a nonzero value in the w component of the vector. This value will be specific to each triangle depending on how it is oriented. And, at long last, the video card hardware can perform interpolation on all the values for our triangles, including colors, texture coordinates, normal vectors, ect. Then when that it is all done, hardware can perform the "perspective divide" resulting in correctly interpolated values for the triangles, along with correctly interpolated attributes.

And that is the real reason everyone everywhere uses 4x4 matrices in hardware and why they are littered throughout API's like OpenGL. Hardware vendors could probably hack their way around some of the math related to translations and transformations but when it comes to projection, you really can't get away from the 4x4 matrix. (but you can sure optimize the heck out of it)

The math for deriving the correct form of a projection isn't to terribly complicated, but the full explanation is fairly long so I will defer to a good book for that, and the best explanation I have seen for deriving a projection matrix that will allow perspective correct interpolation is chapter 6, section 6.2 of the Lengyel book, "Foundations of Game Engine Development".

Answer 2

The bottom row allows you to create perspective foreshortening. That is, it makes lines that are getting further away appear to converge. When arranged this way, we call this a perspective projection matrix.

There are other ways to arrange a projection matrix where that foreshortening doesn't happen. For example in an orthographic projection. This graphic shows some different projections that are possible by changing the values in the last row of a 4x4 matrix.

Mathematically, the difference between a perspective projection and an orthographic projection is that the last row of a perspective matrix has a division in the z component, whereas the orthographic projection has a 0 in the z component.

Answer 3

If we put homogeneous ordinates to the backburner for a second. Then there is also a second reason:

Mathematical completeness. A 3 by 4 or 4 by 3 matrix is not invertible (although that does not mean you couldn't calculate a inverse by other means just not by standard matrix algebra). Inverse is critical for hierarchy calculations.

Now you could posit that the last computer part is always same and be done with it. But having worked for a company that in their main software did just this i would say that it is a mistake. Well, sure now the matrices are invertible but not general. I can now not safely transpose my matrices for example. The truth is that i lose out on a lot of possible mathematics knowledge that i could use because this one part is missing. At the end this was a common source of errors, anger and confusion among clients.

Homogeneous coordinates are just one possible application of this mathematical completeness. It turns out to be very useful if you want to model perspective. But its far from the only benefit.

Answer 4

3x3 matrices cannot represent 3D translations, but 4x4 matrices can

A simple argument why 3D translations are not possible with 3x3 matrices is that translation can take the origin vector:

0
0
0

away from the origin, say to x = 1:

1
0
0

But that would require a matrix such that:

| a b c |   |0|   |1|
| d e f | * |0| = |0|
| g h i |   |0|   |0|

But that is impossible.

A "more overkill"/"more elegant" argument is the singular Value Decomposition theorem, which says that every matrix can be made up with two rotations and one scaling operation:

• rotate to align with eignevectors
• scale with the eigenvectors
• rotate to the final desired position

No translations there.

Why is it possible to use matrices to represent rotations?

Many modeled objects (e.g. a car chassis) or part of modeled objects (e.g. a car tire, a driving wheel) are solids: the distances between vertexes never change.

The only transformations we want to do to on them are rotations and translations.

Matrix multiplication can encode both rotations (NxN matrix) and translations (we need one extra dimension).

Rotation matrices have explicit formulas, e.g.: a 2D rotation matrix for angle a is of form:

cos(a) -sin(a)
sin(a)  cos(a)

There are analogous formulas for 3D, but note that 3D rotations take 3 parameters instead of just 1.

Translations are less trivial and will be discussed later. They are the reason we need 4D matrices.

As a cultural fact, the set of all rotation matrices in a dimension is called the special orthogonal group (SO(n)), which is also a Lie group, and has fundamental applications in particle physics.

Why is it nice to use matrices at all?

Because the composition of multiple matrices can be pre-calculated by matrix multiplication.

E.g., if we are going to translate one thousand vectors v of our car chassis with matrix T and then rotate with matrix R, instead of doing:

v2 = T * v

and then:

v3 = R * v2

for each vector, we can pre-calculate:

RT = R * T

and then do just one multiplication for every vertex:

v3 = RT * v

Even better: if we then want place the vertexes of tire and driving wheel relative to the car, we just multiply the previous matrix RT by the matrix relative to the car itself.

This naturally leads to maintaining a stack of matrices:

• calculate chassis matrix
• multiply by tire matrix (push)
• remove tire matrix (pop)
• multiply by driving wheel matrix (push)
• ...

This stack was explicitly present in the older deprecated immediate mode OpenGL APIs such as glPushMatrix, but with the advent of shaders this responsibility was passed more directly to the shader authors.

How adding one dimension solves the problem

Let's consider the case from 1D to 2D which is easier to visualize.

A matrix in 1D is just one number, and as we've seen in 3D it can't do a translation, only a scaling.

But if we add the extra dimension as we get:

| 1 dx | * |x|  = | x + dx * y |
| 0  1 |   |y|    | y          |

Now, since we added this extra dimension ourselves, we are free to choose a fixed value for y that suites our needs, so we choose y == 1 to get:

| x + dx |
| 1      |

And finally, if we forget about this new extra dimension, we get:

x + dx

as we wanted.

This 2D transformation is so important that it has a name: shear transformation.

It is cool to visualize this transformation:

Image source.

Note how every horizontal line (fixed y) is just translated (larger y being translated more).

We just happened to take the line y = 1 as our new 1D line, and translate it with a 2D matrix.

Things are analogous in 3D, with 4D shear matrices of the form:

| 1 0 0 dx |   | x |   | x + dx |
| 0 1 0 dy | * | y | = | y + dy |
| 0 0 1 dz |   | z |   | z + dz |
| 0 0 0  1 |   | 1 |   |      1 |

And our old 3D rotations / scaling are now of form:

| a b c 0 |
| d e f 0 |
| g h i 0 |
| 0 0 0 1 |

This Jamie King video tutorial is also worth watching.

Affine space

Affine space is the space generated by all our 3D linear transformations (matrix multiplications) together with the 4D shear (3D translations).

If we multiply a shear matrix and a 3D linear transformation, we always get something of the form:

| a b c dx |
| d e f dy |
| g h i dz |
| 0 0 0  1 |

This is the most general possible affine transformation, which does 3D rotation / scaling and translation.

One important property is that if we multiply 2 affine matrices:

| a b c dx |   | a2 b2 c2 dx2 |
| d e f dy | * | d2 e2 f2 dy2 |
| g h i dz |   | g2 h2 i2 dz2 |
| 0 0 0  1 |   |  0  0  0   1 |

we always get another affine matrix of form:

| a3 b3 c3 (dx + dx2) |
| d3 e3 f3 (dy + dy2) |
| g3 h3 i3 (dz + dz2) |
|  0  0  0          1 |

Mathematicians call this property closure, and is required to define a space.

For us, it means that we can keep doing matrix multiplications to pre-calculate final transformations happily, which is why use used matrices in the first place, without ever getting more general 4D linear transformations which are not affine.

Frustum projection

But wait, there is one more important transformation that we do all the time: glFrustum, which makes an object 2x further, appear 2x smaller.

First get some intuition about glOrtho vs glFrustum at: https://stackoverflow.com/questions/2571402/explain-the-usage-of-glortho/36046924#36046924

As a quick reminder, a projection is the act of taking 3D points and plotting them on the 2D camera plane, possibly taking into account perspective if we want to be realistic. This is the key step to translate between the 3D world of vertex shaders and the 2D world of fragment shaders: https://stackoverflow.com/questions/17789575/what-are-shaders-in-opengl-and-what-do-we-need-them-for/36211337#36211337

glOrtho can be done just with translations + scaling , but how can we implement glFrustum with matrices?

Suppose that:

• our eye is at the origin, looking at -z
• the screen (near plane) is at z = -1 that is a square of length 2
• the far plane of the frustum is at z = -2

If only we allowed more general 4-vectors of type:

(x, y, z, w)

with w != 0, and in addition we identify every (x, y, z, w) with (x/w, y/w, z/w, 1), then a frustum transformation with the matrix would be:

| 1 0  0 0 |   | x |   |  x |               | x / -z |
| 0 1  0 0 | * | y | = |  y | identified to | y / -z |
| 0 0  1 0 |   | z |   |  z |               |     -1 |
| 0 0 -1 0 |   | w |   | -z |               |      1 |

If we throw away:

• z because projection means to take a 3D point and place it in a 2D plane
• w because it is just a mathematical calculation artifact we invented

we get:

• x_proj = x / -z
• y_proj = y / -z

which is exactly what we wanted! We can verify that for some values, e.g.:

• if z == -1, exactly on the plane we're projecting to, x_proj == x and y_proj == y.
• if z == -2, then x_proj = x/2: objects are half size.

Note how the glFrustum transform is not of affine form: it cannot be implemented just with rotations and translations.

The mathematical "trickery" of adding the w and dividing by it is called homogeneous coordinates

Image source.

See also: related Stack Overflow question: https://stackoverflow.com/questions/2465116/understanding-opengl-matrices