Sharing code between multiple GLSL shaders

Question

I often find myself copy-pasting code between several shaders. This includes both certain computations or data shared between all shaders in a single pipeline, and common computations which all of my vertex shaders need (or any other stage).

Of course, that's horrible practice: if I need to change the code anywhere, I need to make sure I change it everywhere else.

Is there an accepted best practice for keeping DRY? Do people just prepend a single common file to all their shaders? Do they write their own rudimentary C-style preprocessor which parses #include directives? If there are accepted patterns in the industry, I'd like to follow them.

Answer 1

There's a bunch of a approaches, but none is perfect.

It's possible to share code by using glAttachShader to combine shaders, but this doesn't make it possible to share things like struct declarations or #define-d constants. It does work for sharing functions.

Some people like to use the array of strings passed to glShaderSource as a way to prepend common definitions before your code, but this has some disadvantages:

1. It's harder to control what needs to be included from within the shader (you need a separate system for this.)
2. It means the shader author cannot specify the GLSL #version, due to the following statement in the GLSL spec:

The #version directive must occur in a shader before anything else, except for comments and white space.

Due to this statement, glShaderSource cannot be used to prepend text before the #version declarations. This means that the #version line needs to be included in your glShaderSource arguments, which means that your GLSL compiler interface needs to somehow be told what version of GLSL is expected to be used. Additionally, not specifying a #version will make the GLSL compiler default to using GLSL version 1.10. If you want to let shader authors specify the #version within the script in a standard way, then you need to somehow insert #include-s after the #version statement. This could be done by explicitly parsing the GLSL shader to find the #version string (if present) and make your inclusions after it, but having access to an #include directive might be preferable to control more easily when those inclusions need to be made. On the other hand, since GLSL ignores comments before the #version line, you could add metadata for includes within comments at the top of your file (yuck.)

The question now is: Is there a standard solution for #include, or do you need to roll your own preprocessor extension?

There is the GL_ARB_shading_language_include extension, but it has some drawbacks:

1. It is only supported by NVIDIA (http://delphigl.de/glcapsviewer/listreports2.php?listreportsbyextension=GL_ARB_shading_language_include)
2. It works by specifying the include strings ahead of time. Therefore, before compiling, you need to specify that the string "/buffers.glsl" (as used in #include "/buffers.glsl") corresponds to the contents of the file buffer.glsl (which you have loaded previously).
3. As you may have noticed in point (2), your paths need to start with "/", like Linux-style absolute paths. This notation is generally unfamiliar to C programmers, and means you can't specify relative paths.

A common design is to implement your own #include mechanism, but this can be tricky since you also need to parse (and evaluate) other preprocessor instructions like #if in order to properly handle conditional compilation (like header guards.)

If you implement your own #include, you also have some liberties in how you want to implement it:

• You could pass strings ahead of time (like GL_ARB_shading_language_include).
• You could specify an include callback (this is done by DirectX's D3DCompiler library.)
• You could implement a system that always reads directly from the filesystem, as done in typical C applications.

As a simplification, you can automatically insert header guards for each include in your preprocessing layer, so your processor layer looks like:

if (#include and not_included_yet) include_file();

(Credit to Trent Reed for showing me the above technique.)

In conclusion, there exists no automatic, standard, and simple solution. In a future solution, you could use some SPIR-V OpenGL interface, in which case the GLSL to SPIR-V compiler could be outside of the GL API. Having the compiler outside the OpenGL runtime greatly simplifies implementing things like #include since it's a more appropriate place to interface with the filesystem. I believe the current widespread method is to just implement a custom preprocessor that works in a way any C programmer should be familiar with.

Answer 2

I generally just use the fact that glShaderSource(...) accepts an array of strings as it's input.

I use a json-based shader definition file, which specifies how a shader (or a program to be more correct) is composed, and there I specify the preprocessor defines I may need, the uniforms it uses, the vertex/fragment shaders file, and all the additional "dependency" files. These are just collections of functions that gets appended to the source before the actual shader source.

Just to add, AFAIK, the Unreal Engine 4 uses an #include directive that gets parsed and append all the relevant files, before the compilation, as you were suggesting.

Answer 3

I don't think there is a common convention, but if I'd take a guess, I'd say that almost everyone implements some simple form of textual inclusion as a preprocessing step (an #include extension), because it is very easy to do so. (In JavaScript/WebGL, you can do that with a simple regular-expression, for example). The upside of this is that you can perform the preprocessing in an offline step for "release" builds, when the shader code no longer needs to be changed.

In fact, an indication that this approach is common is the fact that an ARB extension was introduced for that: GL_ARB_shading_language_include. I'm not sure if this became a core feature by now or not, but the extension was written against OpenGL 3.2.

Answer 4

Some people have already pointed out that glShaderSource can take an array of strings.

On top of that, in GLSL the compilation (glShaderSource, glCompileShader) and linking (glAttachShader, glLinkProgram) of shader is separate.

I have used that in some projects to split shaders between the specific part, and the parts common to most shaders, which is then compiled and shared with all shader programs. This works and is not difficult to implement: you just have to maintain a dependency list.

In terms of maintainability though, I am not sure it is a win. The observation was the same, let's try to factorize. While it indeed avoids repetition, the overhead of the technique feels significant. Moreover, the final shader is more difficult to extract: you cannot just concatenate the shader sources, as the declarations end in an order that some compilers will reject, or will be duplicated. So it makes it more difficult to do a quick shader test in a separate tool.

In the end this technique addresses some DRY issues, but it's far from ideal.

On a side topic, I am not sure if this approach has any impact in terms of compilation time; I have read that some drivers only really compile the shader program on linking, but I haven't measured.