Is it possible to set the size of the shared memory of a compute shader workgroup from the host code? There is a way to do this in CUDA and I am suspecting that, since there is an GLSL extension to set the size of a workgroup during launch, there might be a way to do this, as the shared memory size is typically related to workgroup size.
GLSL has no mechanism to change the "size" of shared memory. Indeed, GLSL is structurally incapable of such a thing.
You can't get a pointer to shared memory. Indeed, "shared memory" doesn't really exist in GLSL; there are only shared variables. And variables have a compile-time fixed size. So GLSL wouldn't even have a good way of allowing you to affect how much "shared memory" gets used from outside of the shader.
The closest you might get to this is to have some kind of shared variable array whose size is determined from outside of the shader. But even then, you're not influencing the size of memory in total; you're influencing the size of just that array.
And you can sort of do that. SPIR-V in OpenGL allows specialization constants. These are values specified by the outside world during the compilation process for the shader object. Such constants can be used as the size for a shared variable array. Of course, this means that changing it requires a full recompile/relink process.