# culling meshes across mulitple layers

I am implementing foveated rendering. My FBO is divided into 2 layers which are concentric. The first layer covers 1/3 of the Framebuffer whereas the second one covers the rest of the screen. Now I want to avoid rendering meshes in the outer layer which are already in the inner layer.

I have implemented the following culling algorithm which is only concerned with culling between the layers:

1. I create rectangles with min and max corners from the radii of the 2 layers. Now these coordinates are in NDC.
2. I have the min and max corner of each mesh in the local coordinate system. So I transform them into NDC using:

vec4 min = mvp * vec4(min_corner, 1.0);
min = min.xyzw / min.w;

3. I create a bounding box using min and max corners.

4. I check whether 2 rectangles overlap or not.

Now, when I transform into clip space, the x or y coordinates of the min are sometimes greater than the max corner. The same issue occurs when I divide them by w. What's going wrong here?

EDIT: Okay. I found the problem. Now I iterate over all the vertices, transform each of them into NDC using MVP and then calculate the min and max corners which works perfect. But this is much more overhead for each frame. Is there any optimization I can do here?

• @trichoplax which part of it you did not understand? Commented Feb 4, 2016 at 23:53
• transform bounding sphere radius and center of sphere in 0-1 doesn't make clear whether you want the radius normalised to 1, or transformed to some value between 0 and 1 (in which case, how is that value determined?). The same for the centre - is it to be moved to the origin or translated to somewhere within a distance 1 from the origin or something else? Commented Feb 5, 2016 at 4:56
• somebody pointed out to me that its not possible to transform radius into 0-1 From the comments on your initial question it doesn't appear that anyone is saying that it is not possible to transform the radius. We just need to know what you mean by "0-1". Commented Feb 5, 2016 at 5:00
• @trichoplax okay. I changed my algorithm. Can you check it Commented Feb 5, 2016 at 5:35
• This seems clearer now. I've edited to add in links and hovertext for the acronyms. If this changes your intention please edit to correct this. Commented Feb 6, 2016 at 13:18

## 1 Answer

Regarding optimizations:

Could you calculate the min and max corners in modelspace beforehand? Then you could check on the CPU-side the NDCs of these two points and skip the processing of the whole mesh if it is outside of the desired range. This needs only two matrix-vector multiplications on cpu side. And since for rigid bodies these two points do not change over time, you don't have to iterate the meshes in each frame.

• I can not calculate min max corner in modelspace as after multiplying with view and projection matrices, min and max corner does not remains the same. that is why I am iterating over all the vertices.Previously I thought just using bounding box would suffice but looks like it is changing in clip coordinate space. Commented Feb 8, 2016 at 18:09
• when I checked NDC coordinates at the end in both ways, I get very different numbers. with iteration over vertices I get vertex cooridinates in range of minus thousands to plus thousands whereas with bounding boxes they are in range of -10 plus 10 approx. Commented Feb 9, 2016 at 0:48