2
$\begingroup$

I have some OpenGL based software and I would like to set up some automated testing using image comparison, but one problem we always have is that the screenshots we take always have some small differences across machines, usually due to graphics driver difference or something else (?).

What strategies are there for getting consistent screenshots across machines with varying hardware and graphics drivers?

$\endgroup$
  • $\begingroup$ What sort of differences? $\endgroup$ – gallickgunner Mar 28 '18 at 13:50
  • $\begingroup$ Usually the differences were like little outlines around objects, like if anti aliasing were on one but off on the other. But we turned anti aliasing off always when running the tests. And it didn't always happen, just very rarely with certain objects. $\endgroup$ – default Mar 28 '18 at 16:08
5
$\begingroup$

Nicol Bolas explains why you can't expect pixel-accurate reproductions, but that doesn't mean you can't test with screenshots.

Don't compare the screenshots byte-for-byte. Instead, use a perceptual diff to quantify how different the renders are. You can then set a threshold in your test and have the test fail if the renders are too different. A harsh threshold will let you filter out the rounding differences between platforms, and the slight differences around object silhouettes, but still detect objects in the wrong place or inconsistent texturing. A more forgiving threshold will allow bigger differences in texturing (or even with different shader versions or extensions, if you have different rendering paths for different hardware), and only complain if you have gross differences such as missing lights or a completely different scene being rendered.

The pdiff tool is commonly used for this task, and I've set up test systems based on it before.

$\endgroup$
2
$\begingroup$

What you want cannot reasonably be done. No graphics API defines byte-accurate results. Every implementation will have its own variances, which will be visible in the results. OpenGL provides no insurance that you will get invariant results across implementations.

At the end of the day, you can't do testing in this fashion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.