# Modulo vs Scaling vs Capping when performing filters

I'm currently taking a graphics course and we've recently covered cross-correlation with regards to using a filter matrix that is applied to a region of pixels in a sort of continual raster-type scan across a larger input image to produce some output image.

The formula is denoted here:

G is the output image, h is the kernel/box filter matrix, F is the input image

The classic one that was introduced is a 3x3 box filter for blurring an input image. The box filter is a 3x3 matrix with each cell value being equally set to 1/9. We've been dealing primarily with grayscale images with values 0 (black) to 255 (white).

In this case, the largest possible output pixel value would be if the 3x3 input image area contained all white pixels in which case we'd have:

[(1/9) * 255 ] * 9 = 255

This is fine, but what if our kernel filter matrix instead had 9 cell values all equal to 0.5?

Then the largest potential output value would be:

[0.5 * 255] * 9 = 1147.5

Once we've completely finished producing our new output image matrix, how do we handle values above 255 and below 0?

It seems like there are three possible approaches:

## Capping

Cap all output values to 0 and 255. Any value below 0 is capped to 0, any value above 255 is capped to 255.

## Modulo

Use modulo. In this case it would be 1147.5 % 255 = 125.5 rounded to 126.

## Scaling

Take the highest output value and use it to scale all the values proportionally. If our example output image pixel values were 1147.5, 100, and 255, then we would perform the following operations:

• (1147.5 / 1147.5) * 255 = 255
• (100 / 1147.5) * 255 = 22.2 = 22
• (255 / 1147.5) * 255 = 56.6 = 57

I just don't get which one of these I'm supposed to be using for common graphics operations. I have the same question when combining images together (such as multiply, divide, screen, etc).

When doing a filter operation such as a blur, in many cases the filter itself will be normalized so its values sum to 1.0, precisely to avoid this problem.

In cases where out-of-range outputs do need to be handled, the most common way is what you called capping. In graphics it's more commonly known as clamping or saturating. This is the default behavior if, for instance, you run a pixel shader on a GPU and output out-of-range values.

Renormalizing the output image might be necessary in some special cases depending on what you're doing. I can't think of anytime that I've seen the modulo behavior used—it would be a pretty weird operation to do on a visual image.

• Thanks for this. I was actually working with some images converted to numpy matrices which are uint8 and it automatically uses modulo when you exceed the unsigned byte length so that's why I thought maybe modulo was used in graphics somewhere. Can I ask for clarification on the clamping? So is it common to add/multiply images together? If so, in my samples working with grayscale images it would seem like the final image would just instantly oversaturate and "white-out" so to speak. Is this normal and do we still just clamp the values? Feb 10 '18 at 4:36
• Yes, it's pretty common, though we usually think of images as representing fractional values in [0, 1] rather than integers 0 to 255. Alpha blending, for instance, involves adding and multiplying. Depending on the images and the operations you're doing, saturating to white may indeed happen, but in other cases (like alpha blending layers of images on top of each other) it's not really an issue. Also, if you work with HDR images, you can leave them unclamped while you do a bunch of operations on them, then apply a tone mapping function at the end. Feb 10 '18 at 18:56

While clamping or saturating may be the most common way of handling this situation, it's usually not the best. (But wrapping/modulus is one of the worst.) Another way of handling it is to either convert the input to 16 or 32-bit ints or floats and do all calculations in the higher bit depth. You may want to down-convert the final output, if required by your processing tools. Or, if you can choose the output format you can leave it in a higher bit depth. Most professional tools these days do their work in higher bit depths.