First of all we need to understand why do we need 4x4 matrices in the first place. With 3x3, we couldn't represent translation as it wasn't a linear transformation (it displaces the origin). So in order to avoid extra work, homogeneous coordinates and affine transformation was introduced.
Now instead of doing
$v' = Lv + t$
L is a linear transform and
t is the translation, we can do
$v' = Av$
A is the affine matrix. This makes it cleaner. So 4x4 matrices are a real necessity, we just can't work without them. In order to distinguish between vectors and points we use
w = 1 for points and
w = 0 for vectors. So you are suggesting to make this 4th dimension implicit and don't store it as it'll actually use space/memory.
We can pretend that the 3 dimensional vector is 4 dimensional (imaginary w = 1). This is after all Computer Science, not math.
This works but up-to a certain extent. Using this approach we would have 4x4 matrices and 3 dimensional row/column vectors. But now when doing multiplication we will have to make a check whether it's point or vector and based on that we multiply the elements of the 4th column of the transformation matrix by
This alone starts creating mess, check every time whether it's a point or a vector so why not just use the space for a single float more and get rid of this stupid check.
That aside most modern CPUs have SIMD registers 128 bits wide perfect to fit 4D vectors. If you are doing your calculations on the GPU, all the more reason to store 4D vectors as branching instructions are much more costly on the GPU.
If you start feeling the need to use 4D, great. If not then we still have the problem of composing matrix transformations. Instead of multiplying individual transformations with our vector we would like to pre-multiply all of the transformations then do a single vector-matrix multiplication. This can't work with projective transformations where we need to divide by "something"
Let's come to reason number 3.
You will have to do division by w after projection matrix anyway. Why not divide by z then and get something like [x/z, y/z, z]. Both of them cost 1 computation.
This is the simple case for just a projective transformation. Consider the vector going through 5 or 6 transformations and in the last comes the projective transformation. If we pre-multiply all these transformation to create a single matrix, you will notice that now when we multiply the vector with this combined transformation matrix the division factor isn't just a simple
-z value. The 4th row of the matrix won't be
0 0 -1 0 as in the standard projection matrix. It might have changed due to multiplying all the transformations together. Now when you multiply that 4th row with the 4D vector, you will get your
w value by which you need to divide now.