I'd like to load arbitrary meshes and draw thick black lines along the edges to get a toon-shading like look. I managed to draw a black silhouette around the objects by using the stencil buffer. You can see the result here:

enter image description here

But what's missing are the black lines in the object itself. I thought about checking normal discontinuities: Checking if a neighbouring pixel has a different normal vector than the current one. If yes, an edge has been found. Unfortunately, I've no idea how I could implement this approach, neither in OpenGL nor in GLSL vertex/fragment shader.

I'd be very happy for some help regarding this approach or any other regarding edge detection.

Edit: I do not use any textures for my meshes.

Too be more precise, I would like to create a CAD/CAM solution that looks as much as possible like this (taken from Top Solid https://www.youtube.com/watch?v=-qTJZtYUDB4):

enter image description here

  • $\begingroup$ I believe you need to define the "edge" more in detail. How does it differ from a simple wireframe? In cifz answer there is a good description of screen space post-process, but from your question it is difficult to determine if it is applicable. $\endgroup$
    – Andreas
    Commented May 17, 2016 at 19:07
  • $\begingroup$ Well, with edge I mean the "creases" and "ridges" that form the solids. A wireframe would display all triangle faces, which is not what I want. $\endgroup$
    – enne87
    Commented May 18, 2016 at 7:27
  • $\begingroup$ Ok, exactly what I asked for :-) I would generate a wireframe from those creases and ridges. The tricky part is still to determine what a crease/ridge is. Do you have any idea of how to do that? $\endgroup$
    – Andreas
    Commented May 18, 2016 at 10:44
  • 1
    $\begingroup$ The cad programs do not do this with a shader mostly. Instead they know the hard edges from the model and draw a line on top of the mesh form that info. $\endgroup$
    – joojaa
    Commented May 20, 2016 at 14:51
  • $\begingroup$ joojaa do you have any more information on this technique? What is done in case of double curved free form surfaces? I'm also not sure what happens when you have a cone or a cylinder that is being cut/trimmed by something free form. stackoverflow.com/questions/43795262/… $\endgroup$ Commented Jul 25, 2017 at 20:35

2 Answers 2


Generally edge detection boils down to detect areas of the image with high gradient value.

In our case we can crudely see the gradient as the derivative of the image function, therefore the magnitude of the gradient gives you an information on how much your image changes locally (in regards of neighbouring pixels/texels).
Now, an edge is as you say an indication of discontinuity, so now that we defined the gradient is clear that this info is all we need. Once we find the gradient of an image, it's just a matter of applying a threshold to it to obtain a binary value edge/non-edge.

How do you find this gradient is really what you are asking and I am yet to answer :)

Lots of ways! Here a couple :)

Built in shader functions

Both hlsl and glsl offer derivative functions. In GLSL you have dFdx and dFdy that give you respectively gradient information in x and y direction. Typically these functions are evaluated in a block of 2x2 fragments.
Unless you are interested in a single direction, a good way to have a compact result that indicates how strong is the gradient in the region is fwidth that gives you nothing else but the sum of the absolute value of dFdy and dFdy.
You are likely to be interested in an edge on the overall image rather than on a specific channel, so you might want to transform your image function to luma. With this in mind,when it comes to edge detection your shader could include something to the like of:

  float luminance = dot(yourFinalColour,vec3(0.2126, 0.7152, 0.0722));
  float gradient = fwidth(luminance );
  float isEdge = gradient > threshold;

With an high threshold you will find coarser edges and you might miss some, conversely, with a low threshold you might detect false edges. You have to experiment to find the threshold that better suits your needs.

The reason why these functions work is worth mentioning but I don't have time for it now, I am likely to update this answer later on :)

Screen space post-process

You could go fancier than this, now the field of Edge detection in image processing is immense. I could cite you tens of good ways to detect edge detection according to your needs, but let's keep it simple for now, if you are interested I can cite you more options!

So the idea would be similar to the one above, with the difference that you could look at a wider neighbourhood and use a set of weights on sorrounding samples if you want. Typically, you run a convolution over your image with a kernel that gives you as a result a good gradient info.
A very common choice is the Sobel kernel

                                   enter image description here

Which respectively give you gradients in x and y directions:

                                  From an old pdf I wrote a long time ago.

You can get the single value out of the gradient as $ GradientMagnitude = \sqrt{ (Gradient_x) ^ 2 + (Gradient_y) ^ 2 } $

Then you can threshold as the same way I mentioned above.

This kernel as you can see give more weight to the central pixel, so effectively is computing the gradient + a bit of smoothing which traditionally helps (often the image is gaussian blurred to eliminate small edges).

The above works quite well, but if you don't like the smoothing you can use the Prewitt kernels:

                                                   enter image description here

(Note I am in a rush, will write proper formatted text instead of images soon! )

Really there are plenty more kernels and techniques to find edge detection in an image process-y way rather than real time graphics, so I have excluded more convoluted (pun not intended) methods as probably you'd be just fine with dFdx/y functions.

  • $\begingroup$ Very nice explanation cifz, but what if no gradient is visible in a certain situation? For example, there is no light source at the back of the cube and therefore, no gradient is visible. Then this image based edge detection process you describe wouldn't work, am I right? $\endgroup$
    – enne87
    Commented May 18, 2016 at 7:31
  • $\begingroup$ If you use a deferred renderer you can do the same but on the normal buffer or again, if you have a prepass you could apply the algorithm to the depth. Still you are right, a screen space approach might be not ideal in all cases :) What solution is viable depends a lot on what is your budget for this effect, how complex are your scene. $\endgroup$
    – cifz
    Commented May 19, 2016 at 7:24
  • $\begingroup$ Potentially if this is really central for your game, a very easy, but potentially taxing, would be computing a gradient on your neighboring normals for each vertex (offline at load time) and pass this factor as additional vertex attribute. $\endgroup$
    – cifz
    Commented May 19, 2016 at 7:40
  • $\begingroup$ Thangs again cifz for your help. Well, what I want to achieve is to develop a CAD/CAM solution that looks similar to this: youtube.com/watch?v=-qTJZtYUDB4 And I really would like to know how they managed it to render all the black edges, creases and ridges. cifz, do you think as a graphics programming specialis that they achieved this look by using one of your screen space approaches? Or maybe by computing a gradient on neighboring normals? I really want to know how this can be done. Thanks again! $\endgroup$
    – enne87
    Commented May 19, 2016 at 9:01
  • 1
    $\begingroup$ Don't need to pay anyone, just start reading few tutorials online, buy some books and study hard :) It will be very rewarding at the end! $\endgroup$
    – cifz
    Commented May 20, 2016 at 13:41

Just in case anyone elso also needs to detect edges: Here is a nice article how to display a wireframe and this article explains how to show only the edges.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.