What's the point of "work groups" for OpenGL compute shaders?

Question

glDispatchCompute has the number of work groups as arguments.

What is the point of having work groups? How are they useful, compared to just having NxM independent pixel calculations with a known (x,y) input parameter in the shader?

Answer 1

I won't be discussing the usefulness of workgroups here as you can find it pretty much anywhere. However I am going to touch up on GPU architecture to understand why do we have workgroups in the first place. Most GPUs are designed in such a way that the concept of workgroup is useful. I won't go into details but give a brief overview as to how the mainstream GPUs are designed (AMD/NVIDIA).

So basically both AMD and NVIDIA have hundreds of thousands of cores inside them. AMD groups these cores into what is called compute unit (CU) While NVIDIA groups them to call them a streaming multiprocessor (SM). Here is a picture from AMD architecture as I am more used to it.

As you can see GPU device consists of CU or SM in NVIDIA's case. These then further contain the actual processing elements. AMD has 64 of them for the GCN architecture. Now you are gonna say "so what?" Why do we still have workgroups why don't we use these individual "processing elements" or "cores" by themselves and ditch the concept of workgroup.

Well the thing is, as Matias told above, the compute unit doesn't consist of only these cores, it has much more. The processing elements are just ALUs and they can't do anything on their own. The actual logic that tells them what to do such as the "Branch Unit" the "Instruction Decoding Logic", "Cache" etc are present inside the Compute Unit. Here is another picture for the Compute Unit of GCN Architecture.

So as you see this looks more like a miniature CPU processor with the instruction register and other stuff built in. Here comes the concept of workgroup. In programmer's terms a Workgroup equals or you can say gets assigned to a Compute Unit or thread block equals a Streaming Multiprocessor (in NVIDIA's terms). There are subtle differences such as hardware and software threads which leads to wavefronts and warps but I am not going to touch them here.

So this is somewhat the general architecture how work gets assigned to a GPU. Hence workgroups are necessary. The idea of shared memory and everything Matias said above stems from the underlying GPU architecture.

So when working with images we usually divide it into smaller 2D blocks called workgroups and hand these blocks to get processed by the underlying Compute Units/SMs.

Answer 2

Work groups share resources. The most notable resources they share are barriers and LDS (Local Data Storage aka shared memory in GL lingo, aka Thread Group Shared Memory).

See Efficient Compute Shader Programming (PDF) for an example where LDS/TGSM is used to accelerate a simple gaussian blur.

It is an interesting thought exercise of how a simple algorithm (weight average N values and output the result) can be rewritten to better leverage the hardware. Basically with a kernel radius of 5, Pixel A will load pixels [A; E] to average them, pixel B loads pixels [A; F] to average, pixel C loads pixels [A; G], and so on until we hit pixel F which will load pixels [B; J] to average them. So the paper asks the question: why do we load pixels A through F from main RAM so many times? (note: GPUs DO have caches, but if multiple threads are requesting all the same regions is likely many of these loads will miss the cache or have bank conflicts) and instead tries to load A through F ONCE (each thread loads only one pixel) and store it into TGSM memory which is much faster (and can also be accessed directly by the HW). It's basically very similar to manual cache management.

Also if the workgroup happens to be <= than the GPU's native workgroup size, barriers are much cheaper.

Note not all GPUs are created equal. For example current Mali's GPU do not have LDS, and shared memory is "emulated" via RAM fetches, thus trying to use LDS like a cache is only going to hurt performance.