With respect to this topic I've successfully implemented the Sobel Edge Detection filter in GLSL. Here is the fragment shader code of the filter:

#version 330 core
in vec2 TexCoords;
out vec4 color;

uniform sampler2D screenTexture;

mat3 sx = mat3( 
    1.0, 2.0, 1.0, 
    0.0, 0.0, 0.0, 
   -1.0, -2.0, -1.0 
mat3 sy = mat3( 
    1.0, 0.0, -1.0, 
    2.0, 0.0, -2.0, 
    1.0, 0.0, -1.0 

void main()
    vec3 diffuse = texture(screenTexture, TexCoords.st).rgb;
    mat3 I;
    for (int i=0; i<3; i++) {
        for (int j=0; j<3; j++) {
            vec3 sample  = texelFetch(screenTexture, ivec2(gl_FragCoord) + ivec2(i-1,j-1), 0 ).rgb;
            I[i][j] = length(sample); 

float gx = dot(sx[0], I[0]) + dot(sx[1], I[1]) + dot(sx[2], I[2]); 
float gy = dot(sy[0], I[0]) + dot(sy[1], I[1]) + dot(sy[2], I[2]);

float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));
color = vec4(diffuse - vec3(g), 1.0);

And here is the result of a cube with Sobel edge detection:

Cube with Sobel edge detection filter

If you enlarge the picture, you will see that there is a lot of "noise" produced by Sobel: There are gray horizontal stripes all over the scene due to the blue/white gradient. Futhermore, the light cones produce an unwanted pattern on the cube. The black edges on the left of the cube also seem to fade because of the light cone on the left half of the cube.

So I read this article which stated that one should grayscale the image first and use a Gaussian blur filter to make the edges more apparent. On the bottom of the article, there is also the canny edge detection filter which seem to produce better results.

Now I have two questions:

  1. Are the following steps correct to produce the best possible edge detection results:

    • Grayscale
    • Gaussian Blur
    • Sobel/Canny edge detection
  2. If yes, how would I merge the original image with the processed image? I mean, after processing the steps stated above, I get an image that is either completely black with white edges or vice verca. How would I put the edges on my original image/texture?

Thanks for your help!

  • $\begingroup$ I wonder if you might get the results you want by utilizing OpenGL's edge flag in some way. It looks like only edges between vertices with the edge flag on are drawn when in line mode. There's a description here. (I've not used it myself or I'd post an example.) $\endgroup$ – user1118321 Jun 19 '16 at 21:44
  1. The best results strongly depend on your use case. They also depend on what effect you want to achieve. Sobel is just an edge detection filter: the edges will depend on the input signal, choosing that input signal is up to you.

    Here you are using the color image as an input, and the filter rightfully detects faint edges in the blue gradient, while the edges of the cube get interrupted where its color is too close from the background color.

    Since I assume your program is also responsible for drawing the cube, you have access to other information that you can feed to your Sobel filter. For example depth and normals are good candidates for edge detection. The albedo before lighting could be used to. Test with different inputs and decide which ones to use depending on the results you get.

  2. Regarding your question about how to combine the edge information, I suggest you filter g a little before using it. Then you can use it to interpolate between the original color and the wanted edge color.

    For example you could try something like this:

    float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));

    // Try different values and see what happens
    g = smoothstep(0.4, 0.6, g);

    vec3 edgeColor = vec3(1., 0., 0.2);
    color = vec4(mix(diffuse, edgeColor, g), 1.);


To use depth or normals, you would need to save them in a texture if that's not done already. When you create the frame buffer for your regular rendering pass, you can attach various textures to it (see glFramebufferTexture2D) and write other information to them than just the color of the scene.

If you attach a depth texture (with GL_DEPTH_ATTACHMENT), it will be used automatically for depth. If you attach one or several color textures (with GL_COLOR_ATTACHMENTi), you can write to them by declaring several outputs to your fragment shaders (this used to be done with gl_FragData; either way, see glDrawBuffers).

For more information on the topic, look up "Multi Render Target" (MRT).

  • $\begingroup$ Nice information, thanks Julien. One more question: How could I use the depth or normal information in my fragment shader? I'm still quite new with GLSL and therefore have no idea how to include the values in the Sobel algorithm. $\endgroup$ – enne87 Jun 19 '16 at 14:25
  • 2
    $\begingroup$ When you create the framebuffer for your rendering pass, you can attach more than one texture to it. If you attach a depth texture, it will be used automatically for depth. It you attach another color texture you can write into it separately (see opengl.org/sdk/docs/man/html/glDrawBuffers.xhtml), a feature called "Multi Render Target" (MRT). $\endgroup$ – Julien Guertault Jun 20 '16 at 0:12
  • $\begingroup$ @enne87: Happy to help. :) $\endgroup$ – Julien Guertault Jun 21 '16 at 2:52
  • $\begingroup$ @trichoplax: thanks for the suggestion; done. $\endgroup$ – Julien Guertault Jun 22 '16 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.