I'm currently taking a graphics course and we've recently covered cross-correlation with regards to using a filter matrix that is applied to a region of pixels in a sort of continual raster-type scan across a larger input image to produce some output image.
G is the output image, h is the kernel/box filter matrix, F is the input image
The classic one that was introduced is a 3x3 box filter for blurring an input image. The box filter is a 3x3 matrix with each cell value being equally set to 1/9. We've been dealing primarily with grayscale images with values 0 (black) to 255 (white).
In this case, the largest possible output pixel value would be if the 3x3 input image area contained all white pixels in which case we'd have:
[(1/9) * 255 ] * 9 = 255
This is fine, but what if our kernel filter matrix instead had 9 cell values all equal to 0.5?
Then the largest potential output value would be:
[0.5 * 255] * 9 = 1147.5
Once we've completely finished producing our new output image matrix, how do we handle values above 255 and below 0?
It seems like there are three possible approaches:
Cap all output values to 0 and 255. Any value below 0 is capped to 0, any value above 255 is capped to 255.
Use modulo. In this case it would be 1147.5 % 255 = 125.5 rounded to 126.
Take the highest output value and use it to scale all the values proportionally. If our example output image pixel values were 1147.5, 100, and 255, then we would perform the following operations:
(1147.5 / 1147.5) * 255 = 255
(100 / 1147.5) * 255 = 22.2 = 22
(255 / 1147.5) * 255 = 56.6 = 57
I just don't get which one of these I'm supposed to be using for common graphics operations. I have the same question when combining images together (such as multiply, divide, screen, etc).