Is sub-texel shading possible?

Question

I suspect the answer is no, but thought I'd ask anyway.

It seems that other than using nearest neighbour and bilinear interpolations, there is nothing you can do to determine how two adjacent pixels are interpolated ... such as applying an edge preserving interpolation, or adding some procedural variation.

I'd love to be wrong! If indeed there is no way to do this, might there be a practical solution which involves creating a higher resolution (maybe much higher) texture (like a "mipmap level -2") on the fly?

If done naively I guess it would be very memory-hungry ... perhaps there is some way it could be done to a reusable buffer, or some way ensuring only part of the image would be processed at a time (like an OpenCL/CUDA/Compute Shader workgoup) thus limiting the buffer size. (Perhaps Vulkan could help somehow?)

Obviously it would be done on a LOD basis ... no need to do it on far away textures.

Thanks

Answer 1

The GPU hardware only supports nearest-neighbor, bilinear, trilinear, and anisotropic texture filtering. However, nothing stops you from implementing your own texture filtering in the pixel shader.

To do this, you'd use textureGather to grab a 2×2 block of texels around your sample point. Then you'd manually calculate the position of the sample point relative to the texel centers. Armed with the raw texels and the interpolation position, you can then apply whatever interpolation function you like.

The position calculation would be something like this:

vec2 textureSize; // texture's width and height
vec2 offset = fract(texCoord * textureSize + (-0.5 + 1.0/512.0));

The 1/512 offset in there is needed as a rounding correction on certain GPUs, to ensure that fract()'s output steps from 1 to 0 at the exact same point that textureGather switches texels.

As an example, to re-implement bilinear filtering, you'd do this (assuming a single-channel texture for simplicity):

vec4 texels = textureGather(...);
float result = mix(mix(texels.w, texels.z, offset.x),
                   mix(texels.x, texels.y, offset.x),
                   offset.y);

But in practice, you'd be replacing the above with your own interpolation logic. This will for sure be slower than using hardware bilinear interpolation, but it may be an acceptable cost for your app.

If you need more than 2×2 block of texels, you can also use multiple calls to textureGatherOffset to grab more of the surrounding region, 2×2 at a time.

---

As for generating a higher-resolution version of the texture on the fly, that can certainly also be done. It would be an extension of texture streaming techniques that load larger mip levels from disk only when needed. Here, you'd be procedurally generating the larger mip levels instead of loading them, but it's a similar idea. Unfortunately I don't have a good reference handy for how to do this. (It's somewhat related to virtual texturing, but less complicated—for just plain texture streaming, you don't need sparse textures or a page table, etc.)

However, there is unfortunately no GPU hardware support for doing the thing you suggested of avoiding the extra storage, by processing the texture only a piece at a time. Doing that in hardware (“texel shaders” that would generate texel values on-demand as they are sampled by fragment shaders) is an idea that has been proposed many times but so far has not happened.

It might be possible to do it in software by having fragment shaders append sample requests into a queue, then switching to a different shader to service those requests, then re-running the fragment shaders with the results...but that would take a lot of work and might not be very fast. It would for sure be a research project. 😄

Unless you have a good way of determining in advance which texels are likely to be needed, it would be a lot more practical to just generate the entire texture, assuming memory and performance budgets allow for it.