1
$\begingroup$

I'm re-writing an algorithm that I first wrote using matrix/vector operation into OpenGL kernels in order to try to maximiz the performances.

I have a basic knowledge of OpenGL, so I was able to get the things working, but I have a lot of trouble when it come to make various choices offered by OpenGL, especially the buffer's parameters that I guess have a huge impact in my case where I read and write a lot of data.

I call the three kernels sequencially :

First :

/* Generated constants (for all three shaders): 
 *   #version 430
 *   const vec3 orig
 *   const float vx
 *   const ivec2 size
 *   const uint projections
 *   const uint subIterations
 */
layout(local_size_x = 1, local_size_y = 1) in;

layout(std430, binding = 0) buffer bufferA { //GL_SHADER_STORAGE_BUFFER, GL_DYNAMIC_READ
    uint bufferProjection[]; //Written and read (AtomicAdd) by this shader, read by the second kernel
};
layout(std430, binding = 1) readonly buffer bufferB { //GL_SHADER_STORAGE_BUFFER, GL_DYNAMIC_READ
    uint layer[]; //Written and read by the third kernel, read by this shader and by glGetNamedBufferSubData
};
layout(std140) uniform bufferMat { //GL_UNIFORM_BUFFER, GL_STATIC_DRAW
    mat4 proj_mat[projections*subIterations]; //Read only by this shader and the third
};
layout(location = 0) uniform int z;
layout(location = 1) uniform int subit;

void main() {
    vec4 layer_coords = vec4(orig,1.0) + vec4(gl_GlobalInvocationID.x, z, gl_GlobalInvocationID.y, 0.0)*vx;
    uint val = layer[gl_GlobalInvocationID.y*size.x + gl_GlobalInvocationID.x];
    for(int i = 0; i < projections; ++i) {
        vec4 proj_coords = proj_mat[subit+i*subIterations]*layer_coords;
        ivec2 tex_coords = ivec2(floor((proj_coords.xy*size)/(2.0*proj_coords.w)) + size/2);
        bool valid = all(greaterThanEqual(tex_coords, ivec2(0,0))) && all(lessThan(tex_coords, size));
        atomicAdd(bufferProjection[tex_coords.y*size.x+tex_coords.x+i*(size.x*size.y)], valid?val:0);
    }
}

Second:

layout(local_size_x = 1, local_size_y = 1) in;

layout(std430, binding = 0) buffer bufferA { //GL_SHADER_STORAGE_BUFFER, GL_DYNAMIC_READ
    float updateProjection[]; //Written by this shader, read by the third kernel
};
layout(std430, binding = 1) readonly buffer bufferB { //GL_SHADER_STORAGE_BUFFER, GL_DYNAMIC_READ
    uint bufferProjection[]; //Written by the first, read by this shader
};
layout(std430, binding = 2) readonly buffer bufferC { //GL_SHADER_STORAGE_BUFFER, GL_DYNAMIC_READ
    uint originalProjection[]; //Only modified by glBufferSubData, read by this shader
};

void main() {
    for(int i = 0; i < projections; ++i) {
        updateProjection[gl_GlobalInvocationID.x+i*(size.x*size.y)] = float(originalProjection[gl_GlobalInvocationID.x+i*(size.x*size.y)])/float(bufferProjection[gl_GlobalInvocationID.x+i*(size.x*size.y)]);
    }
}

Third:

layout(local_size_x = 1, local_size_y = 1) in;

layout(std430, binding = 0) readonly buffer bufferA { //GL_SHADER_STORAGE_BUFFER, GL_DYNAMIC_READ
    float updateProjection[]; //Written by the second kernel, read by this shader
};
layout(std430, binding = 1) buffer bufferB { //GL_SHADER_STORAGE_BUFFER, GL_DYNAMIC_READ
    uint layer[]; //Written and read by this shader, read by the first kernel and by glGetNamedBufferSubData
};
layout(std140) uniform bufferMat { //GL_UNIFORM_BUFFER, GL_STATIC_DRAW
    mat4 proj_mat[projections*subIterations]; //Read only by this shader and and the first
};
layout(location = 0) uniform int z;
layout(location = 1) uniform int subit;
layout(location = 2) uniform float weight;

void main() {
    vec4 layer_coords = vec4(orig,1.0) + vec4(gl_GlobalInvocationID.x, z, gl_GlobalInvocationID.y, 0.0)*vx;
    float acc = 0;
    for(int i = 0; i < projections; ++i) {
        vec4 proj_coords = proj_mat[subit+i*subIterations]*layer_coords;
        ivec2 tex_coords = ivec2(floor((proj_coords.xy*size)/(2.0*proj_coords.w)) + size/2);
        bool valid = all(greaterThanEqual(tex_coords, ivec2(0,0))) && all(lessThan(tex_coords, size));
        acc += valid?updateProjection[tex_coords.y*size.x+tex_coords.x+i*(size.x*size.y)]:0;
    }
    float val = pow(float(layer[gl_GlobalInvocationID.y*size.x + gl_GlobalInvocationID.x])*(acc/projections), weight);
    layer[gl_GlobalInvocationID.y*size.x + gl_GlobalInvocationID.x] = uint(val);
}

What I came up by reading the OpenGL doc :

  • Some value that are the same for all the duration of the algorithm are generated as const before compiling the shader. Especially usefull for the for-loop boundary
  • bufferMat, which is very small compared to the other buffers, is put in an UBO, which should have better performance than SSBO. Do I could get event better perf by making it a compile-time constant ? It's small, but still a few hundreds mat4
  • The others buffer, being both read and written several times, should be better as SSBO
  • I have trouble understanding what could be the best value for the 'usage' parameters of the buffer. All buffers being written to and read several time, I'm not sure what to put here.
  • If I understand correctly, the local_size is only usefull when sharing data between invocations, so I should keep it at one ?

I would gladly take any recommendations or hints about where to look to optimize thoses kernels !

$\endgroup$
1
$\begingroup$

Do I could get event better perf by making it a compile-time constant ?

You'll have to profile it. That being said, "a few hundreds mat4" is not "small".

I have trouble understanding what could be the best value for the 'usage' parameters of the buffer. All buffers being written to and read several time, I'm not sure what to put here.

First, the usage parameters are about your usage of the buffer object, not OpenGL's usage of the memory behind them. That is, they're talking about functions like glBufferSubData, glMapBufferRange, and so forth. READ means that the CPU will read from the buffer, but not write to it. DRAW means that the CPU will write to the buffer, but not read from it.

Second... you really shouldn't care. Usage hints are terrible, poorly specified, and have been so misused that many implementations flat-out ignore them. NVIDIA's GL implementation is probably the one that takes them the most seriously.

Instead, use immutable storage buffers. Those "usage hints" are not hints; they're API requirements. If you don't use GL_DYNAMIC_STORAGE_BIT, then you cannot write to the buffer via glBufferSubData. And so forth.

If I understand correctly, the local_size is only usefull when sharing data between invocations, so I should keep it at one ?

No. In fact, never use 1. If all of the invocations are doing their own thing, with no execution barriers or the like, then you should pick a local size that is equivalent to the wavefront size of the hardware you're working on. Obviously that's implementation dependent, but 32 is certainly a better default than 1.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks a lot, that was exactly this kind of informations I was looking for ! After a bit of search and tweakings my buffers and my work group size, I got a 10x speed bump ! $\endgroup$ – 04348 Aug 20 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.