1
$\begingroup$

There is this article about conversions in Direct3D: https://docs.microsoft.com/en-us/windows/win32/direct3d10/d3d10-graphics-programming-guide-resources-data-conversion

I do not understand the part of conversion from higher to lower precision:

Round-to-zero is used during conversion to another float format. If the target is an integer or fixed point format, round-to-nearest-even is used, unless the conversion is explicitly documented as using another rounding behavior, such as round-to-nearest for FLOAT to SNORM, FLOAT to UNORM or FLOAT to SRGB.

What does round-to-zero mean, e.g. if I want to convert float -> half? What does round-to-nearest-even mean, e.g. for half->unorm?

Why is one round-to-nearest-even while another conversion is round-to-nearest?

Phrased differently, how do these conversions work?

$\endgroup$
2
$\begingroup$

Rounding is all about what happens to the data in the higher precision format that don't exist in the lower precision one. For example, 0.7 cannot exactly be represented as an 8-bit unsigned, normalized integer, as 0.7 * 255 = 178.5. Which gets stored: 178 or 179?

The same goes for any other precision. 16-bit floats can't store as many digits as a 32-bit float.

So when doing rounding, you (conceptually) first convert the number with full precision, then you figure out which of the two valid values in the destination precision are available, and then you use an algorithm to pick one of these two.

Round-to-nearest-even means that you round to the closest of the two values, but if you're halfway between them, you pick the even number of the two. So if we're converting 178.5 into a non-normalized integer, 178.5 is halfway between 178 and 179. But 178 is an even number, so we pick that instead of the traditional method of rounding up halves.

Round-to-zero means that you always pick the number which is closest to zero, no matter how close you are to the other one.

Round to zero is used for float-to-float conversions because that is what happens if you just drop the extra bits of the mantissa in the converted value. Because IEEE-754 floats are stored as a magnitude + sign-bit, the magnitude doesn't contain a sign value. So if you remove significant low precision bits, the resulting number is closer to zero.

As to why there's a difference when converting to normalized and non-normalized integers, that's unclear.

$\endgroup$
1
  • $\begingroup$ Thanks for that. While I now understand how Round-to-nearest-even works, is there any benefit to doing it this way? On first glance, it looks like this is a very inconsistent way of rounding, seeing as both 177.5 and 178.5 would round to 178. It's a bit hard to describe, but it is like the ranges become (-0.5, 0.5) around odd numbers and [-0.5, 0.5] around even numbers, which seems... I don't know, "unmathematic"? $\endgroup$
    – Tare
    Nov 24 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.