Incorrect values written in stencil buffer

Question

I am trying to pick objects on mouse click. For this I have followed this tutorial, and tried to use the stencil buffer for this purpose.

Inside "game" loop I am trying to draw 10 (5 pairs) 'pick'able triangles as follows:

...

glClearColor(red, green, blue, 1.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glClearStencil(0); // this is the default value

/* Enable stencil operations */
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

/*Some other drawing not involving stencil buffer*/

GLuint index = 1;
for (GLshort i = 0; i < 5; i++)
{
    //this returns 2 model matrices
    auto modelMatrices = trianglePairs[i].getModelMatrices(); 

    for (GLshort j = 0; j < 2; j++)
    {
        glStencilFunc(GL_ALWAYS, index, -1);
        glUniformMatrix4fv(glGetUniformLocation(ourShader.Program, "model"), 1, GL_FALSE, glm::value_ptr(modelMatrices[j]));
        glDrawElements(GL_TRIANGLES, 3, GL_UNSIGNED_INT, BUFFER_OFFSET(2));
        index++;
    }
    /*Some other drawing not involving stencil buffer*/
}

/*Some other drawing not involving stencil buffer*/
...

However, when I am trying to read back the stencil values, I am getting wrong values. I am reading back the values as (this is also a part of the above-mentioned tutorial):

GLuint index;
glReadPixels(xpos, Height - ypos - 1, 1, 1, GL_STENCIL_INDEX, GL_UNSIGNED_INT, &index);

Whenever, I click the first triangle of the pair I am getting values as i+1, whereas the correct value should have been i, and for the second triangle of the pair, I am getting 0 as index.

Please let me know what am I missing here?

Update I have found that stencil values can be applied on quads. When I tried to apply the stencil value on rectangle it worked correctly. Though it is not the solution I was looking at, still, I can live with that. However, I would still like to know what was the problem with triangles.

Answer 1

OK, so this not really an answer, rather just sharing what I have learned.

To clarify, shape of the object has nothing to do with this problem at all. Be it a triangle or rectangle, and wherever you click on that object, this method should work (the 'But' for this part is explained below).

I think stencil buffer values are associated with each pixel on screen, and it does not directly maps to the OpenGL window space. Initially I was using a window size that is smaller than my screen size and on top of that it was not full screen. Each time I started (debugging from VS 2015), the window appeared not exactly at (0, 0) on the screen. Some of the screen space is also consumed by the window bar at the top. I think this is the reason for getting erratic values when reading back the data from stencil buffer.

So as a simple solution to this problem what I did is to make the window full screen, and everything worked like charm. As I use GLFW, here is how I have done it:

glfwCreateWindow(Width, Height, "Title", glfwGetPrimaryMonitor(), nullptr);

Just used glfwGetPrimaryMonitor(), while creating the window. And that's it.

And yes, it works for triangles too.