I'm using OpenGL's stencil test to carve out a semicircular hole in a transition frustum with a cylinder oriented perpendicularly. However as one can see below, while this objective is achieved, the cylinder is still visible through the frustum which shouldn't be the case.

enter image description here

I'm using the following code fragment that I found at https://en.wikibooks.org/wiki/OpenGL_Programming/Mini-Portal

glStencilFunc(GL_NEVER, 1, 0xFF);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // draw 1s on test fail (always)

// draw stencil pattern
glClear(GL_STENCIL_BUFFER_BIT);  // needs mask=0xFF

// draw where stencil's value is 0
glStencilFunc(GL_EQUAL, 0, 0xFF);
/* (nothing to draw) */
// draw only where stencil's value is 1
glStencilFunc(GL_EQUAL, 1, 0xFF);



Moreover I've found that the results are insensitive to the parameters of the glStencilOp and DepthMask commands.

I seem to vaguely remember from several years ag that a multi-pass rendering may be required. If so, could someone point me to how to accomplish this with the proper set of command parameters?

thanks in advance.

This is an example of what I'm expecting - The cylinder should cut a hole through the frustum. I've rendered it here by trimming the black cylinder within the frustum so that it just just out past the frustum surface.

enter image description here

  • 1
    $\begingroup$ Dan, I've added an image describing my expectation as an "answer". $\endgroup$ Jun 29, 2017 at 16:28
  • 1
    $\begingroup$ You probably want to search for CSG (Constructive Solid Geometry) and Stencils. For example, this usenix.org/legacy/events/usenix05/tech/freenix/full_papers/… might allow you to achieve the desired results. Similarly, perhaps look at opencsg.org ? Actually, I might add this as a possible answer if no other answers appear $\endgroup$
    – Simon F
    Jun 29, 2017 at 16:43
  • 1
    $\begingroup$ Thanks! In fact, I was reading that paper last night, but was hoping that OpenGL would have provided a way of accomplishing this with just basic stencil functions. It looks like quite an involved task! $\endgroup$ Jun 29, 2017 at 16:50
  • 1
    $\begingroup$ Simon, one of the issues that I'd have with using the EXCELLENT CSG resources is that I'm accessing OpenGL via Visual Basic modules linked to an Excel spreadsheet from which I drive my application. I do not know whether or not the project files for MSVC6 will include extensions for VBA. $\endgroup$ Jun 29, 2017 at 17:49
  • 1
    $\begingroup$ I´d treat the cylinder as a "light source", putting its geometry as a shader uniform and test fragment position against the cylinder, clipping fragments inside. Not sure if it is feasible for you or even interesting but in case you or others are stuck I´m just saying there is a way :-) $\endgroup$
    – Andreas
    Sep 13, 2017 at 18:08

1 Answer 1


When cylinder and frustum intersect the hole will be a view into the internals of the cylinder. The hole is where external of the cylinder occlude frustum and internal of the cylinder does not occlude the frustum.

Knowing this you should simply do the following to find the surface (or rather volume) made from the hole.

Render frustum without color and set depth accordingly. Set a stencil bit where the cylinder (external) occludes the frustum, without setting new depth. Clear the previously set bit where the internal of the cylinder occludes the frustum, without setting new depth.

Alternate between internal and external using CullFace.

The stencil mask you now have is the surface where the frustum hole is. Render the frustum (in color) masking it with the stencil. Make sure to use LEQUALS depth test to allow the frustum to render on top of itself. Voila. Note that the depth buffer holds the value of the unmasked frustum so If you need to render something within the hole you would have to render the whole scene again and mask it to only render in the hole unless depth test passes against the frustum. Left as student exercise.

  • $\begingroup$ Sorry for this late acknowledgement, but I had to shelve this project owing to other more pressing priorities, but now that I've some time to return to it, I'll investigate your suggestion. Thanks a ton! $\endgroup$ Apr 10, 2018 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.