Perspective projection matrices we use map 3D coords onto some imaginary image plane in front of the cameras pinhole position by casting a ray directly from the eye position to that 3d coordinate and seeing where it intersects the image plane.
But if you take your average 4x4 perspective projection matrix (assume no reversed z or infinite far plane at the moment), where exactly is this thing? As in, if you drew a diagram, where would you draw this "world space image plane" and all the contents in it?
It must be one of the planes orthogonal to the view direction. All points projected onto any of these planes will produce the same image only differing in scale. Yet presumably in one of them the ray along the top left of the view frustum of the camera will strike the image plane at the coordinate [-1,1] (depending on your choice of clip space coordinates), and the depth value of this intersection is disregarded (instead the distance of the point from the camera receives a custom remapping function for various reasons) and so on for the other frustum edges.
But I'm having a hard time figuring out how this can be with cameras that don't have a 1:1 aspect ratio. If I'm drawing a classical view frustum on paper and check some intersections on the frustum corners, it's never at [x,y] with |x| = |y|.