I have a problem with precision on AMD in shaders (hlsl). In vertex shader I calculate UVs for particle using modulus operator on float. With simple % there is a problem on Nvidia and on AMD (e.g. 7 % 7 = 7) so I have written modulus as:

float Mod( float x, float y )    
     return x - floor(x / y) * y;    

This have fixed the problem on Nvidia but AMD still returns wrong values.

Any idea what is going on? And how to fixed it?

I know I can use:

float( int( x ) % int( y ) )

but I think it is slower and I wonder if there is any better way.

  • $\begingroup$ Are you using DX or are you using vulkan? Both support HLSL... $\endgroup$ – opa May 30 '18 at 19:58
  • $\begingroup$ I'm using DirectX11 $\endgroup$ – Derag Jun 13 '18 at 21:16

In HLSL, you can use the built-in fmod function for this.

I'm not sure what's going wrong with your implementation, though. It looks mathematically correct. It's conceivable you're encountering a precision issue with the subtraction, depending on the inputs.

BTW, an alternative implementation which avoids the subtraction is:

float Mod(float x, float y)
    return frac(x / y) * y;
  • $\begingroup$ But frac(x / y) * y is exactly what normal modulus operator compiles to (on GCN at least) and it gives incorrect results on both Nvidia and AMD. I'm not sure about fmod, though. It seems to give slightly different assembly code. I will check it, thanks. $\endgroup$ – Derag Apr 19 '18 at 19:45
  • $\begingroup$ So I have chacked it and frac(x / y) * y, operator % and fmod give incorrect results on Nvidia and AMD. x - floor(x / y) * y works correct on Nvidia but AMD not so much. $\endgroup$ – Derag Apr 20 '18 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.