# culling meshes across mulitple layers

I am implementing foveated rendering. My FBO is divided into 2 layers which are concentric. The first layer covers 1/3 of the Framebuffer whereas the second one covers the rest of the screen. Now I want to avoid rendering meshes in the outer layer which are already in the inner layer.

I have implemented the following culling algorithm which is only concerned with culling between the layers:

1. I create rectangles with min and max corners from the radii of the 2 layers. Now these coordinates are in NDC.
2. I have the min and max corner of each mesh in the local coordinate system. So I transform them into NDC using:

vec4 min = mvp * vec4(min_corner, 1.0);
min = min.xyzw / min.w;

3. I create a bounding box using min and max corners.

4. I check whether 2 rectangles overlap or not.

Now, when I transform into clip space, the x or y coordinates of the min are sometimes greater than the max corner. The same issue occurs when I divide them by w. What's going wrong here?

EDIT: Okay. I found the problem. Now I iterate over all the vertices, transform each of them into NDC using MVP and then calculate the min and max corners which works perfect. But this is much more overhead for each frame. Is there any optimization I can do here?

• @trichoplax which part of it you did not understand? – debonair Feb 4 '16 at 23:53
• transform bounding sphere radius and center of sphere in 0-1 doesn't make clear whether you want the radius normalised to 1, or transformed to some value between 0 and 1 (in which case, how is that value determined?). The same for the centre - is it to be moved to the origin or translated to somewhere within a distance 1 from the origin or something else? – trichoplax Feb 5 '16 at 4:56
• somebody pointed out to me that its not possible to transform radius into 0-1 From the comments on your initial question it doesn't appear that anyone is saying that it is not possible to transform the radius. We just need to know what you mean by "0-1". – trichoplax Feb 5 '16 at 5:00
• @trichoplax okay. I changed my algorithm. Can you check it – debonair Feb 5 '16 at 5:35
• This seems clearer now. I've edited to add in links and hovertext for the acronyms. If this changes your intention please edit to correct this. – trichoplax Feb 6 '16 at 13:18