Is it possible to reduce aliasing in derivatives used to estimate normals from a texture?

Question

I am generating a normal map for a texture for use in a PBR shader. I have a map for the diffuse texture, and I am estimating the normal at each point using the derivative of the texture's brightness. I store the result in a new texture and read the values out of it as a standard normal map.

This is the method used by tools such as SmartNormap. It works quite well and I am happy with the quality of the result.

However, there is a caveat. In the same shader, I am also using a hex tiling algorithm to generate an infinitely repeating, seamless output. I currently run this algorithm on both the diffuse map as well as the normal map. Since it is an expensive algorithm, this is causing a significant perf impact.

To try to mitigate the perf impact and avoid the double lookups, I had the idea of generating the normal map on the fly within the fragment shader based on derivatives of the fragment colors. I have created an implementation that looks something like this:

float diffuseMagnitude = length(diffuseColor);
float dDiffuseX = dFdx(diffuseMagnitude);
float dDiffuseY = dFdy(diffuseMagnitude);

float computeNormalBias = 0.1;
vec3 mapN = normalize(vec3(
  dDiffuseX,
  dDiffuseY,
  1.0 - ((computeNormalBias - 0.1) / 100.0)
));
mapN.xy *= normalScale;

normal = perturbNormal2Arb( - vViewPosition, normal, mapN, faceDirection );

Here is what the results looks like for that.

Before:

After:

You may have to expand the images to see the impact in more detail.

This does work and I find it to be a great improvement, but there is a rather severe issue. Since these derivatives are computed in screen space, the world space distance between pixels can become quite large at higher distances and sharp angles of incidence with the camera. This results in severe aliasing when moving in certain directions.

Demo video: https://ameo.link/u/bcafbfde12486f95388b5c448db2c1dbb037f9b4.mp4

---

I think I understand the reason this issue issue is happening, but I don't know how to solve it - or if it's even possible to solve it.

Normally, I'd solve aliasing by oversampling, but there's no way to really oversample when using the dFdx/dFdy functions that I know of.

I know that my use of that expensive anti-tiling algorithm really limit things but it's very important to my use case and I can't do without it. If I am unable to find a solution with this dynamic normal generation idea, I will probably go back to pre-generating normal maps and doing the double lookups and then turning it off in low detail mode or something similar.

If anyone has any ideas or pointers, though, I'd love to hear them!