Generally edge detection boils down to detect areas of the image with high gradient value.
In our case we can crudely see the gradient as the derivative of the image function, therefore the magnitude of the gradient gives you an information on how much your image changes locally (in regards of neighbouring pixels/texels).
Now, an edge is as you say an indication of discontinuity, so now that we defined the gradient is clear that this info is all we need. Once we find the gradient of an image, it's just a matter of applying a threshold to it to obtain a binary value edge/non-edge.
How do you find this gradient is really what you are asking and I am yet to answer :)
Lots of ways! Here a couple :)
Built in shader functions
Both hlsl and glsl offer derivative functions. In GLSL you have dFdx and dFdy that give you respectively gradient information in x and y direction. Typically these functions are evaluated in a block of 2x2 fragments.
Unless you are interested in a single direction, a good way to have a compact result that indicates how strong is the gradient in the region is fwidth that gives you nothing else but the sum of the absolute value of dFdy and dFdy.
You are likely to be interested in an edge on the overall image rather than on a specific channel, so you might want to transform your image function to luma. With this in mind,when it comes to edge detection your shader could include something to the like of:
float luminance = dot(yourFinalColour,vec3(0.2126, 0.7152, 0.0722));
float gradient = fwidth(luminance );
float isEdge = gradient > threshold;
With an high threshold you will find coarser edges and you might miss some, conversely, with a low threshold you might detect false edges. You have to experiment to find the threshold that better suits your needs.
The reason why these functions work is worth mentioning but I don't have time for it now, I am likely to update this answer later on :)
Screen space post-process
You could go fancier than this, now the field of Edge detection in image processing is immense. I could cite you tens of good ways to detect edge detection according to your needs, but let's keep it simple for now, if you are interested I can cite you more options!
So the idea would be similar to the one above, with the difference that you could look at a wider neighbourhood and use a set of weights on sorrounding samples if you want. Typically, you run a convolution over your image with a kernel that gives you as a result a good gradient info.
A very common choice is the Sobel kernel
Which respectively give you gradients in x and y directions:
You can get the single value out of the gradient as $ GradientMagnitude = \sqrt{ (Gradient_x) ^ 2 + (Gradient_y) ^ 2 } $
Then you can threshold as the same way I mentioned above.
This kernel as you can see give more weight to the central pixel, so effectively is computing the gradient + a bit of smoothing which traditionally helps (often the image is gaussian blurred to eliminate small edges).
The above works quite well, but if you don't like the smoothing you can use the Prewitt kernels:
(Note I am in a rush, will write proper formatted text instead of images soon! )
Really there are plenty more kernels and techniques to find edge detection in an image process-y way rather than real time graphics, so I have excluded more convoluted (pun not intended) methods as probably you'd be just fine with dFdx/y functions.
