What I managed to figure out is that the first 3*3 is used for rotation and scale and the 1*3 at the end is used for position, but what is the bottom row used for? is it only for clipping related things?
The bottom row allows you to create perspective foreshortening. That is, it makes lines that are getting further away appear to converge. When arranged this way, we call this a perspective projection matrix.
There are other ways to arrange a projection matrix where that foreshortening doesn't happen. For example in an orthographic projection. This graphic shows some different projections that are possible by changing the values in the last row of a 4x4 matrix.
Mathematically, the difference between a perspective projection and an orthographic projection is that the last row of a perspective matrix has a division in the z component, whereas the orthographic projection has a 0 in the z component.
If we put homogeneous ordinates to the backburner for a second. Then there is also a second reason:
Mathematical completeness. A 3 by 4 or 4 by 3 matrix is not invertible (although that does not mean you couldn't calculate a inverse by other means just not by standard matrix algebra). Inverse is critical for hierarchy calculations.
Now you could posit that the last computer part is always same and be done with it. But having worked for a company that in their main software did just this i would say that it is a mistake. Well, sure now the matrices are invertible but not general. I can now not safely transpose my matrices for example. The truth is that i lose out on a lot of possible mathematics knowledge that i could use because this one part is missing. At the end this was a common source of errors, anger and confusion among clients.
Homogeneous coordinates are just one possible application of this mathematical completeness. It turns out to be very useful if you want to model perspective. But its far from the only benefit.