Facetted look when using SampleBias

Question

I am writing a simple material system for a game. Each material can have a specular value (phong) that is translated into "smoothness" via environment map.

Similar to PBR, I compute a fresnel term that modulates the "smoothness", to have reflections at gazing angles.

The problem is that I seem to get a facetted look when I use the HLSL function SampleBias to modulate the mipmap selection in regards to the fresnel term.

My shader looks roughly like this:

float roughness;
float fresnel = pow(1 - saturate(dot(-viewDir, normal)), 3);
roughness = saturate(1 - saturate(s * specularBoost));
float environmentBias = roughness * 10; // 10 is the number of mipmaps for a 512x512px texture
float3 r = normalize(reflect(viewDir, normal));
float m = 2. * sqrt(
pow(r.x, 2.) +
pow(r.y, 2.) +
pow(r.z + 1., 2.)
);
float2 uv = r.xy / m + 0.5f; // environment map is a sphere map, not a cube/equirectangular map
float4 environmentMap = environment.SampleBias(texSampler, uv,environmentBias);

When I use SampleBias I receive a facetted look (vertex normals are 100% shared and normalized in the Pixel shader!)
It looks like the UVs are shifted across the polygons

When I use SampleLevel directly, I receive a smooth look:

I have no clue what I am doing wrong.

This is roughness:

calculated UVs:

ddx(uv)*0.5+0.5 and ddy(uv)*0.5+0.5 look like a constant yellow to me

Edit:

here is the whole thing with different colored mip maps.
SampleBias:

SampleLevel: