4
$\begingroup$

I'm currently taking a graphics course and we've recently covered cross-correlation with regards to using a filter matrix that is applied to a region of pixels in a sort of continual raster-type scan across a larger input image to produce some output image.

The formula is denoted here:

G is the output image, h is the kernel/box filter matrix, F is the input image

The classic one that was introduced is a 3x3 box filter for blurring an input image. The box filter is a 3x3 matrix with each cell value being equally set to 1/9. We've been dealing primarily with grayscale images with values 0 (black) to 255 (white).

In this case, the largest possible output pixel value would be if the 3x3 input image area contained all white pixels in which case we'd have:

[(1/9) * 255 ] * 9 = 255

This is fine, but what if our kernel filter matrix instead had 9 cell values all equal to 0.5?

Then the largest potential output value would be:

[0.5 * 255] * 9 = 1147.5

Once we've completely finished producing our new output image matrix, how do we handle values above 255 and below 0?

It seems like there are three possible approaches:

Capping

Cap all output values to 0 and 255. Any value below 0 is capped to 0, any value above 255 is capped to 255.

Modulo

Use modulo. In this case it would be 1147.5 % 255 = 125.5 rounded to 126.

Scaling

Take the highest output value and use it to scale all the values proportionally. If our example output image pixel values were 1147.5, 100, and 255, then we would perform the following operations:

  • (1147.5 / 1147.5) * 255 = 255
  • (100 / 1147.5) * 255 = 22.2 = 22
  • (255 / 1147.5) * 255 = 56.6 = 57

I just don't get which one of these I'm supposed to be using for common graphics operations. I have the same question when combining images together (such as multiply, divide, screen, etc).

$\endgroup$
2
$\begingroup$

When doing a filter operation such as a blur, in many cases the filter itself will be normalized so its values sum to 1.0, precisely to avoid this problem.

In cases where out-of-range outputs do need to be handled, the most common way is what you called capping. In graphics it's more commonly known as clamping or saturating. This is the default behavior if, for instance, you run a pixel shader on a GPU and output out-of-range values.

Renormalizing the output image might be necessary in some special cases depending on what you're doing. I can't think of anytime that I've seen the modulo behavior used—it would be a pretty weird operation to do on a visual image.

$\endgroup$
  • $\begingroup$ Thanks for this. I was actually working with some images converted to numpy matrices which are uint8 and it automatically uses modulo when you exceed the unsigned byte length so that's why I thought maybe modulo was used in graphics somewhere. Can I ask for clarification on the clamping? So is it common to add/multiply images together? If so, in my samples working with grayscale images it would seem like the final image would just instantly oversaturate and "white-out" so to speak. Is this normal and do we still just clamp the values? $\endgroup$ – zoombini Feb 10 '18 at 4:36
  • $\begingroup$ Yes, it's pretty common, though we usually think of images as representing fractional values in [0, 1] rather than integers 0 to 255. Alpha blending, for instance, involves adding and multiplying. Depending on the images and the operations you're doing, saturating to white may indeed happen, but in other cases (like alpha blending layers of images on top of each other) it's not really an issue. Also, if you work with HDR images, you can leave them unclamped while you do a bunch of operations on them, then apply a tone mapping function at the end. $\endgroup$ – Nathan Reed Feb 10 '18 at 18:56
2
$\begingroup$

While clamping or saturating may be the most common way of handling this situation, it's usually not the best. (But wrapping/modulus is one of the worst.) Another way of handling it is to either convert the input to 16 or 32-bit ints or floats and do all calculations in the higher bit depth. You may want to down-convert the final output, if required by your processing tools. Or, if you can choose the output format you can leave it in a higher bit depth. Most professional tools these days do their work in higher bit depths.

Regarding your question:

is it common to add/multiply images together?

Yes, it's quite common in tools that allow compositing. In Photoshop, for example, there are blend modes that are even named "Add" and "Multiply" and they do just that. (Though they assume the values are normalized to a 0-1 range.) If you are working with 8-bit per channel images, they will clamp, but if you are working with 32-bit per channel images, they will simply have values greater than 1. If done with linear RGB, this mimics light fairly naturally, and you can apply natural photographic operations to the results, like decreasing the exposure and it will bring those blown-out highlights back into the displayable range. As we move into the era of High Dynamic Range output devices, this type of processing will only become more common and useful.

$\endgroup$
  • $\begingroup$ Thanks! Normalization into 0-1 floating point values makes sense to me because then multiplying won't cause the threshold to be exceeded. What would you do in the case of adding/subtracting images together? Would you clamp the values between 0-1 after the arithmetic? $\endgroup$ – zoombini Feb 10 '18 at 15:41
  • $\begingroup$ Yes, once you've completed all the operations you need to on the image, only then do you clamp it. So if you're adding 2 images, then blurring the result, or scaling it, or whatever, you do all of those things at the higher bit depth and then clamp the final output (if needed). $\endgroup$ – user1118321 Feb 10 '18 at 15:43
  • $\begingroup$ I guess this doesn't work for grayscale, right? Let's say I have a standard 8-bit RGB image, 8-bits for each channel. Image 1 is a solid red color image where every pixel is [200, 0, 0]. Image 2 is a solid dark red color image where every pixel is [100, 0, 0]. When I up-convert to a higher bit depth, do I literally just use 16-bits for each channel, so my final image would be an image where every pixel is now [300, 0, 0]? Now my choices are (A) Leave it at this 16-bit per channel depth or (B) Convert back to 8-bit in which case the image will clamp back to [255, 0, 0]. Is that right? $\endgroup$ – zoombini Feb 10 '18 at 15:53
  • $\begingroup$ This absolutely works for grayscale. It also works for other color spaces like HSV, YCbCr, Lab, etc. And yes, you have the right idea about what I'm suggesting. $\endgroup$ – user1118321 Feb 10 '18 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.