I am trying out different anti-aliasing algorithms on my image to improve the image quality. I want to compare the differences between these generated images via some metric. I was checking online for good metrics and found none.
Could someone please help me by pointing me to a valid metric that one can use to measure aliasing effects in an image ?
What you want to do is compare your anti aliased results against a "ground truth" image.
The ground truth image is the same image but without aliasing.
There are many ways to make an alias-free image, but a straight forward way would be by doing super sampling.
Super Sampling
When we render, we usually take a single sample at the center of the pixel (rasterize / shader / etc it to make the color).
Super sampling works by taking $N$ samples for that pixel, in random parts of the pixel and averaging the result.
The more samples you take, the closer you get to the ideal "alias free" image.
One simple way to do super sampling is to render the image at a higher resolution and then shrink it down to a smaller image. For instance, if you render at double the height and width and then shrink the image down, that's the same as taking 4 samples per pixel. However, going this route, the samples are evenly spaced on a grid, not randomly placed within each pixel. Making the samples random instead of evenly spaced is really important to this monte carlo integration working, but evenly spacing them on a grid is a cheaper method that is better than nothing (aka useful in realtime situations sometimes).
Another way to do super sampling would be to render the image at normal resolution $N$ times, but in each render, you randomly offset the camera by a sub-pixel amount. You then average your $N$ images together. That will give you the random sample points, but of course each pixel has the same random sample points.
This is why you will often see sub-pixel jittering of the camera when doing temporal anti aliasing: it allows for better anti aliased results (a better integration) over time when combining multiple frames. If they were all in the same position every frame, you wouldn't get any more information about the pixel.
Doing these methods will give you your ground truth alias free image, if using enough samples.
Comparing vs Ground Truth
The next step would be to compare the other anti aliased techniques against the super sampled ground truth.
There are lots and lots of ways that you might want to analyze this data, but I'll give a few ideas.
One way could be to make a texture where each pixel is the absolute value of the difference between the images for that pixel. This would let you see the entire image as a whole and get an idea of how different it was from the ground truth.
Another way could be to calculate the mean and standard deviation of the difference of the images. This would give you an idea of how much on average pixels differed in the images, while also giving you an idea of how much variance there was in the difference. A high variance means some places are much worse than other places, so isn't very consistent. Ideally, the best AA method will have the lowest mean, and the lowest variance.
Lastly, since aliasing is ultimately a "frequency" thing, you might find interesting data in doing a discrete Fourier transform on your images and comparing the frequencies in the AA method images vs the frequencies in the super sampled image.
More info on taking DFT of images here: https://blog.demofox.org/2016/07/28/fourier-transform-and-inverse-of-images/
