What causes blobby edges with alpha testing?

Question

I'm trying to understand what causes the blobby edges when using alpha testing to create transparency with 1-bit alpha channels. The edges appear pixelated, but with rounded corners and diagonal lines. [Image from http://wiki.polycount.com/wiki/File:Alphatest_8bit_vs_1bit.jpg]

My theory is that the fragment shader is sampling from the alpha channel using bicubic interpolation then comparing with the alpha threshold. (I would expect straighter edges with bilinear interpolation). Is this correct? Is something special done to prevent overshoot artifacts?

Answer 1

Let's consider the simplest case, where our texture is 2x2, with alpha values $a_{0,0}, a_{1,0}, a_{0,1}, a_{1,1}$. With bilinear filtering, the alpha at $uv$ coordinate $(x,y)$ is

lerp(lerp($a_{0,0}$,$a_{1,0}$,$x$),lerp($a_{0,1}$,$a_{1,1}$,$x$),$y$),

which works out to:

lerp($x a_{1,0} + (1-x) a_{0,0}$, $x a_{1,1} + (1-x) a_{0,1}$,$y$)

= $y(x a_{1,1} + (1-x) a_{0,1}) + (1-y) (x a_{1,0} + (1-x) a_{0,0})$,

Expanding this out, it's a polynomial of the form:

$b+cx+dy+exy$.

Solving when this is equal to some threshold generally produces hyperbolas, but in degenerate cases can produce lines.

In the image below, the texture specifies the colors at each grid line intersection. Within each grid square we have an example of interpolating a 2x2 texture, and the edge produced by comparing with a threshold is a segment of line or hyperbola.

I had originally assumed the lerp of the lerps would produce a polynomial of the form $b+cx+dy$, which solving against a threshold can only produce lines, so it was surprising to see nonlinear curves in the result.