Is it better to blur the input or output of an edge detection shader for noise reduction?

Question

To reduce noise of edge detection the norm seems like it is to apply a blur. However, is it generally better to apply the blur to the input of the edge detection. The input in my case being the depth and normal GBuffers in which I compare neighbouring pixels.
Or is it better to blur the output of the edge detection, e.g. a greyscale image of edge strength, before I use that to apply some edge to a colour buffer?

Answer 1

Standard blur removes high frequency content from the signal, whereas edge detection usually look into high frequency to detect edges. Be careful on how much blurring to apply to ensure that you don't lose desirable edges. The goal of blurring is to perform noise reduction, so the best would be to come up with a model of the noise present in your images and what your desired edges looks like (their frequency, orientation, what caused them: shading on a smooth object, shadows, etc.) and tailor your blur filter to remove the noise but not the edges.

Seems like you want to implement an edge detector into a shader. Be sure to check what are the sources of noise in your image. For example, if you are displaying pure lambertian surfaces, chances are that there is virtually no noise in your data, so you don't need to apply a smoothing filter before detecting edges.

Also, instead of using a standard gaussian blur, you could look into edge-preserving filters like the bilateral filter. It is much slower, but is good to remove noise without altering too much the edges.

If you are performing filtering and edge detection through convolutions (the standard way of doing it), it doesn't matter if you do one before another, as convolutions are commutative, meaning $a*b = b*a$, where $*$ is the convolution operator, $a$ being the edge detector and $b$ the smoothing kernel.

On non-associative operations (like the bilateral filter), you must ask yourself the question "what is the goal of this operation?" Basically, blurring before the edge detection will diminish the image noise while blurring after will give you blurred edges (potentially caused by noise) instead of sharp edges. The same edges will be found, though, but the blurred one will be less precise. It depends on what you want to achieve: it could be interesting for art, but it just removes interesting information for signal analysis.

Not sure what "applying an edge to a color buffer" means; is the goal to perform edge enhancement?