How to dynamically calculate texture coordinates in a vertex shader with OpenGL?

Question

I modified my code, and tried to pass a set of integers specifying the face of a cube (top, bottom left...) to the vertex shader in order to draw multiple instances of a cube face.

I did this by trying to pass them as vertex attributes, but it isn't working as I expected, the value of the int should be between 0 and 5, but it's never a number between 0 and 5, in fact it seems to be much much larger than that.

This is my current shader:

#version 450

#define PI 3.1415926535897932384626433832795

layout(location = 0) in vec3 position; //(x,y,z) coordinates of a vertex
layout(location = 1) in vec3 norm; //a 3D vertex representing the normal to teh vertex 
layout(location = 2) in vec2 texture_coordinate;
layout(location = 3) in int face_type;

out vec3 normal; 
out vec3 vertexPos; //projected vertex
out vec2 texture_coord;

uniform mat4 model = mat4(1); //Position and orientation of the current object
uniform mat4 view = mat4(1); //Camera orientation and position
uniform mat4 proj = mat4(1); //the projection parameters (FOV, viewport dimensions)

//Taken from: https://gist.github.com/neilmendoza/4512992
mat4 rotationMatrix(vec3 axis, float angle)
{
    axis = normalize(axis);
    float s = sin(angle);
    float c = cos(angle);
    float oc = 1.0 - c;

    return mat4(oc * axis.x * axis.x + c,           oc * axis.x * axis.y - axis.z * s,  oc * axis.z * axis.x + axis.y * s,  0.0,
                oc * axis.x * axis.y + axis.z * s,  oc * axis.y * axis.y + c,           oc * axis.y * axis.z - axis.x * s,  0.0,
                oc * axis.z * axis.x - axis.y * s,  oc * axis.y * axis.z + axis.x * s,  oc * axis.z * axis.z + c,           0.0,
                0.0,                                0.0,                                0.0,                                1.0);
}

void main()
{
    mat4 rotation = mat4(1);
    switch(face_type)
    {
        case 0:
            rotation = mat4(1);
            break;
        case 1:
            rotation = rotationMatrix(vec3(0,0,1), -PI/2.f);
            break;
        case 2:
            rotation = rotationMatrix(vec3(0,0,1), PI/2.f);
            break;
        case 3:
            rotation = rotationMatrix(vec3(0,0,1), PI);
            break;
        case 4:
            rotation = rotationMatrix(vec3(1,0,0), PI/2.f);
            break;
        case 5:
            rotation = rotationMatrix(vec3(1,0,0), -PI/2.f);
            break;

        default:
            rotation = mat4(vec4(face_type,0,0,0),
                            vec4(0,1,0,0),
                            vec4(0,0,1,0),
                            vec4(0,0,0,1));
            break;
    }

    /*rotation = mat4(vec4(5,0,0,0),
                    vec4(0,5,0,0),
                    vec4(0,0,5,0),
                    vec4(0,0,0,1));*/

    vec4 projection = proj*view*model*rotation*vec4(position, 1.0);
    gl_Position = projection;

    normal = vec3(model*vec4(norm,1.0));
    vertexPos = vec3(model*vec4(position, 1.0)); //calculate the transformed pos
    texture_coord = texture_coordinate + vec2(face_type*(1/6.f),0);
}

And this is how I am passing the face information to the shader:

vector<Face> face_types = {Top, Bottom, Left,};
glEnableVertexAttribArray(3);
glBindBuffer(GL_ARRAY_BUFFER, experiment);
glVertexAttribPointer(3, 1, GL_INT, GL_FALSE, sizeof(Face), (void*)0);
glBufferData(GL_ARRAY_BUFFER, face_types.size()*sizeof(Face),
    face_types.data(), GL_DYNAMIC_DRAW);
glVertexAttribDivisor(3, 1);

Answer 1

To be able to give a good answer, we need to know what is the compiler error you are referring to.

At first sight your shader looks ok, although:

• It always sets texture_coord.y to 0, which might not be what you want.
• The .f notation (in orientation*(1/6.f)) doesn't exist in GLSL. It should be 1./6.. Moreover, I am not entirely sure if strict GLSL allows to multiply a uint with a float; a cast might be necessary.
• You also need to make sure that on the receiving end, the fragment shader is declaring the texture coordinate as needed: in vec2 texture_coord;.

It is fairly common to set texture coordinates dynamically in the vertex shader, so you should be able to do so too.

---

Update to your additional questions and details:

OpenGL throws an INVALID_OPERATION error

OpenGL error handling is quite tricky, since you have to poll for errors yourself, and it uses the same few error codes for wildly different functions. You have to identify exactly which function yielded the error, then refer to the documentation for the list of reasons this error can happen for that function. To make that task easier, the best way I have found so far is to always surround OpenGL calls with a macro doing an error check. You can find an implementation of a macro like that in the source code of bgfx for example.

The error comes from my face type array being too big, but now I don't know how to pass the face type information into the shader, since I know I need 2048 elements at most

The layout declared in the shader must match the one declared through the OpenGL calls. The number of vertices is irrelevant on the shader side, but it must be consistent on the OpenGL declaration side. Unless you are using different indexes for the vertex attributes, you should have as many positions, normals, texture coordinates and face types.

The following snippet from your question seems to be declaring face types for only three vertices though:

glBindBuffer(GL_ARRAY_BUFFER, experiment);
glVertexAttribPointer(3, 1, GL_INT, GL_FALSE, sizeof(Face), (void*)0);
glBufferData(GL_ARRAY_BUFFER, face_types.size()*sizeof(Face), face_types.data(), GL_DYNAMIC_DRAW);
glVertexAttribDivisor(3, 1);

I am not familiar with glVertexAttribDivisor, but from what I read, it seems to be limited to instanced rendering.

You can find a simple example of binding in the examples of Dear ImGui.