1
$\begingroup$

I've been working on a 2d shadow system for a school project, using this method to generate visibility polygons. However, I'm having issues figuring out how to draw those visibility polygons in a way that reveals everything drawn within them, rather than drawing over them like a normal sprite would.

From what I've been able to figure out, it seems like what I want to do is to draw the visibility polygons to some texture first, and then use that texture somehow when drawing the rest of the objects on the screen to get the shadow/light effect.

Unfortunately all the tutorials I've been able to find aren't very helpful. So, my questions are:

1) How do I tell OpenGL to render to a texture instead of the screen?

2) How do I use that texture to create the effect I want?

Improve this question
$\endgroup$
3
  • $\begingroup$ this is somewhat the same question or atleast the answer is the same as this. the approach is essentially the inverse of the answer you got. Apparently the examples have undergone link rot so i need to see if i can find an alternate solution for this. $\endgroup$
    – joojaa
    Commented Aug 31, 2017 at 17:49
  • $\begingroup$ With one lightsource they may be similar but that approach wouldn't work for multiple light sources, which is something I want to be able to do. $\endgroup$
    – AgentPaper
    Commented Sep 1, 2017 at 18:22
  • $\begingroup$ It would work with as many light sources as you like. Just render multiple times. $\endgroup$
    – joojaa
    Commented Sep 1, 2017 at 21:57

1 Answer 1

Reset to default
3
$\begingroup$

You don't need to render to texture to achieve an effect like this. Use stencil buffer.

The visibility polygon can be drawn using GL_TRIANGLE_FAN, if you start from the middle:

mask

At the beginning of each frame:

  1. Clear the stencil buffer (and enable stencil test)
  2. Set the stencil function to always pass, and to replace the current value in the buffer to a greater than zero number (for example one)
  3. Draw the visible part as a GL_TRIANGLE_FAN. If you have multiple characters/lights, draw the visibility polygon for each.

Now the stencil buffer contains 1 at visible locations and 0 where the character can't see.

Once you have a stencil buffer like this, you can control which objects you want to draw using glStencilFunc. Do not write to the stencil buffer, just use the stencil test to discard fragments which are visible/invisible.

If you want to draw a background which is visible everywhere, use GL_ALWAYS. For objects which are only displayed in the visible area use GL_EQUAL and 1. Similarly you can draw something (for example a dark quad) just in the invisible parts of the scene with GL_EQUAL and 0.

If it is not clear, how to use stencil buffers, check out this tutorial

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks! I had seen that along the way, but thought it was for something else. One question though, is a stencil buffer all or nothing, or would it be possible to have some parts be more/less visible based on ie: distance from light source, overlapping light sources. $\endgroup$
    – AgentPaper
    Commented Aug 30, 2017 at 7:50
  • 1
    $\begingroup$ The value (in case of a 8bit stencil) can be anything between 0-255. There are also glStencilOp's like GL_INCR and GL_DECR. If 0=hidden, 1=half-visible, 2=visible, then you can draw a black quad on the screen with GL_EQUAL and 0, and a half-transparent gray quad with GL_EQUAL and 1. If you would like to draw continuous shadow gradients, then you have to use a shadow texture. But stencil buffers are still really useful even if you have a shadow texture, because the stencil test is much cheaper than a discard in the fragment shader $\endgroup$
    – Iter Ator
    Commented Aug 30, 2017 at 9:00
  • $\begingroup$ Hm, so sounds like I definitely want a stencil buffer to mark off everything that's not lit up at all. However I would definitely like to be able to have gradient shadows or possibly even tinted shadows for various effects. So then I'd need a texture buffer for that? Or a frame buffer? Or are those the same thing? $\endgroup$
    – AgentPaper
    Commented Aug 31, 2017 at 0:27
  • $\begingroup$ If you render to texture, you will use a frame buffer with only one color attachment. A framebuffer is an object, which contains depth+stencil buffer and color attachment(s). You already use the default framebuffer to render to the screen. But if you would like to render offscreen (to a texture) you have to create a new one for it. The texture is the color attachment in that case. I advise you to implement it with stencil buffers only, and if everything looks as intended, start experimenting with textures $\endgroup$
    – Iter Ator
    Commented Aug 31, 2017 at 9:10
  • $\begingroup$ Manged to get the stencil version up and running, everything looks pretty good. Added checks for winding on the line segments so you can see the object casting the shadow as well, which makes for a nice effect. How do I tell OpenGL which frame buffer to draw to? Do I need a different shader program for that, or do I just use the same one like for stencil drawing? And for stencil drawing, would it be more efficient to use a specific program that doesn't take color data for that draw call, or does that not matter? $\endgroup$
    – AgentPaper
    Commented Sep 2, 2017 at 3:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.