I've heard a lot about jittering and dithering but I would like to know more about those techniques, especially when used to avoid visible sampling in a fragment shader.
Jittering and dithering are both techniques of adding noise to reduce visible artefacts (such as banding) in an image. They solve different kinds of artefacts so they are used in different situations. Jittering moves sample positions in space to reduce artefacts caused by regular sampling. Dithering changes the way colours are rounded (when reducing precision), replacing banding with noise.
A use case for jittering is when generating eye rays in a ray-tracer. Without jittering, you might generate one ray in the centre of each pixel. But then if you trace those at a regular pattern, such as a grille of bars each one pixel wide, you might find that every ray hits a bar, making the grille solid; or that every ray hits a gap, and the grille isn't seen at all.
Instead, you can add a small stochastic displacement to each ray direction, so that each ray goes through a different part of its pixel. That way, some rays will hit the bars of the grille and some will hit the gaps, making a noisy image instead of a completely wrong image.
The normal use of dithering is when rounding colours (intensity values) from a high-precision format (such as floating-point numbers) to a low-precision format (such as 8-bit integers). Say you have a smooth gradient you want to represent in an 8-bit image. If you round each number separately, then starting from the black end of the gradient, you get a lot of black pixels until they get light enough to round to 1/256; then these all have the same value until you get to 2/256; and so on. You end up with regular bands where all the pixels have the same value.
Instead, when you round the first number down, you can let it "donate" the lost intensity to another nearby pixel. That way, then at the black end of the gradient, maybe every tenth pixel is 1/256 and the other nine are 0. The ratio of black to nearly-black pixels increases gradually, until eventually all the pixels are 1/256, and then every tenth is 2/256, and so on. Instead of visible bands, the overall intensity increases smoothly along the gradient - but the price of this is that pixels which should be the same are slightly different, adding noise.
Dithering is used a lot in printing, where the colour resolution is quite poor (i.e. the steps between colours are big) but the spatial resolution (how small you can make each dot of colour) is very fine. It used to be used in the days of 256-colour displays for making colours which couldn't otherwise be represented, and it's still used to avoid banding when displaying (for example) a 16-bit-per-channel image on an 8-bit display.
Because traditional dithering algorithms involve giving information from each pixel to its neighbours, they don't parallelise well, so they're not suited to modern hardware. Another popular technique is to deliberately add noise to the source image. The range of the noise is less than one step of the destination format (e.g. the noise ranges from -1/512 to 1/512), so it can't be seen directly, but it means that each pixel will be rounded differently. Like true dithering, it replaces any banding with subtle noise in the output image.
There's no reason you can't use both in the same image, but as they solve different problems, that's not really "combining" them. It's like asking if we can combine hoses and axes. A fire truck has both, and the firefighters might use both at the same incident, but one is for extinguishing fires and the other is for freeing trapped people.
An example of when you might use both is if you ray-trace an image into a floating-point framebuffer, using jittered ray positions to avoid sampling artefacts, and then reduce that to an 8-bit image using dithering to better preserve the colours.
