culling meshes across mulitple layers

Question

I am implementing foveated rendering. My FBO is divided into 2 layers which are concentric. The first layer covers 1/3 of the Framebuffer whereas the second one covers the rest of the screen. Now I want to avoid rendering meshes in the outer layer which are already in the inner layer.

I have implemented the following culling algorithm which is only concerned with culling between the layers:

1. I create rectangles with min and max corners from the radii of the 2 layers. Now these coordinates are in NDC.

2. I have the min and max corner of each mesh in the local coordinate system. So I transform them into NDC using:

   vec4 min = mvp * vec4(min_corner, 1.0);
   min = min.xyzw / min.w;
   

3. I create a bounding box using min and max corners.

4. I check whether 2 rectangles overlap or not.

Now, when I transform into clip space, the x or y coordinates of the min are sometimes greater than the max corner. The same issue occurs when I divide them by w. What's going wrong here?

EDIT: Okay. I found the problem. Now I iterate over all the vertices, transform each of them into NDC using MVP and then calculate the min and max corners which works perfect. But this is much more overhead for each frame. Is there any optimization I can do here?

Answer 1

Regarding optimizations:

Could you calculate the min and max corners in modelspace beforehand? Then you could check on the CPU-side the NDCs of these two points and skip the processing of the whole mesh if it is outside of the desired range. This needs only two matrix-vector multiplications on cpu side. And since for rigid bodies these two points do not change over time, you don't have to iterate the meshes in each frame.