# Incorrect values written in stencil buffer

I am trying to pick objects on mouse click. For this I have followed this tutorial, and tried to use the stencil buffer for this purpose.

Inside "game" loop I am trying to draw 10 (5 pairs) 'pick'able triangles as follows:

...

glClearColor(red, green, blue, 1.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glClearStencil(0); // this is the default value

/* Enable stencil operations */
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

/*Some other drawing not involving stencil buffer*/

GLuint index = 1;
for (GLshort i = 0; i < 5; i++)
{
//this returns 2 model matrices
auto modelMatrices = trianglePairs[i].getModelMatrices();

for (GLshort j = 0; j < 2; j++)
{
glStencilFunc(GL_ALWAYS, index, -1);
glDrawElements(GL_TRIANGLES, 3, GL_UNSIGNED_INT, BUFFER_OFFSET(2));
index++;
}
/*Some other drawing not involving stencil buffer*/
}

/*Some other drawing not involving stencil buffer*/
...


However, when I am trying to read back the stencil values, I am getting wrong values. I am reading back the values as (this is also a part of the above-mentioned tutorial):

GLuint index;
glReadPixels(xpos, Height - ypos - 1, 1, 1, GL_STENCIL_INDEX, GL_UNSIGNED_INT, &index);


Whenever, I click the first triangle of the pair I am getting values as i+1, whereas the correct value should have been i, and for the second triangle of the pair, I am getting 0 as index.

Please let me know what am I missing here?

Update I have found that stencil values can be applied on quads. When I tried to apply the stencil value on rectangle it worked correctly. Though it is not the solution I was looking at, still, I can live with that. However, I would still like to know what was the problem with triangles.

• Keep in mind that most stencil buffers only have 8-bits allotted to them, which limits the number of unique IDs to 256. A similar strategy that allows for a much greater number of items is color picking. With it you render each object using a basic flat shader and a unique RGB color value. You then check your render target for the color under the mouse click. Here is an example (have not done more than a cursory look over the article). – ssell Jun 22 '16 at 19:38
• @ssell Thank you for your comment. I have also seen that article when I started working on object selection. As I am a newbie in OpenGL, I resorted to the stencil buffer approach, which I think is pretty straightforward, and also fits my need as I have only 10 objects to pick. I have found a (buggy) workaround to this problem, I will share the details soon (by weekend I guess:P). – Sayan Pal Jun 22 '16 at 19:46
• My OpenGL is rusty, but everything seems to be set up OK. Keep in mind that an index of 0 indicates your background (an area where no triangle was drawn) as that is the default stencil clear value when calling glClear. Do you happen to have an image of the scene you are drawing? An accompanying render of the stencil buffer would also be useful. Also keep in mind that you may need to invert your y value when performing the actual picking if your window coordinate system does not match your texture coordinate system (OpenGL origin is in lower-left corner, Windows origin is in upper-left). – ssell Jun 22 '16 at 19:59
• Are you actually making these triangles visible? If not, could you draw them? The fact that quads are returning a non-zero stencil value just makes me think you should check you haven't got a winding order problem with the second triangle of each pair. – Simon F Jun 24 '16 at 9:02

OK, so this not really an answer, rather just sharing what I have learned.

To clarify, shape of the object has nothing to do with this problem at all. Be it a triangle or rectangle, and wherever you click on that object, this method should work (the 'But' for this part is explained below).

I think stencil buffer values are associated with each pixel on screen, and it does not directly maps to the OpenGL window space. Initially I was using a window size that is smaller than my screen size and on top of that it was not full screen. Each time I started (debugging from VS 2015), the window appeared not exactly at (0, 0) on the screen. Some of the screen space is also consumed by the window bar at the top. I think this is the reason for getting erratic values when reading back the data from stencil buffer.

So as a simple solution to this problem what I did is to make the window full screen, and everything worked like charm. As I use GLFW, here is how I have done it:

glfwCreateWindow(Width, Height, "Title", glfwGetPrimaryMonitor(), nullptr);


Just used glfwGetPrimaryMonitor(), while creating the window. And that's it.

And yes, it works for triangles too.