Your actual result is correct in terms of what you can achieve in a single-pass with a fragment shader. It calculates the distance of the fragment's world position to the line segment. That distance is different for each fragment so the stretched black ellipsoid is the correct output.
A fragment shader has only local information about the current fragment being processed. You can't do more with that. Your expected result requires some global knowledge about the scene, since you are searching for the minimum distance across the whole surface.
Without any thoughts on optimization, the naive solution path should involve contextual computation of local minima of the fragment-to-linesegment distance. In order to get such context, you need to compute the distance field first and then access it to find the local minima in a separate pass.
- Compute the raw distance field in the first pass (same as your example but no
radius subtraction at the end.
- Mark the pixels with local minima in the second pass (e.g. by setting the alpha channel to 1). This will be screen-space pass. The best is to have a
rgba float texture at output with
rgb being fragment's world position and
a being 1 for local minima and
- Paint the black circles/spheres in a third pass (also screen-space).
Having the information about the local minimum of your distance function enables you to output the right shape for around the minima in the last pass. In pseudocode:
vec4 worldPos = tex2D(distance_texture, frag.x, frag.y);
if (worldPos.a > 0.5) //is minimum
//Sample the screen-space neighborhood in larger and larger circles
// until a fragment with distance.a == 1 is found
// or until all fragments are more than radius away (in world coords).
//Minimum world space distance of any sampled fragment.
float minDist = 0.0;
int iter = 1;
//Search for a local minimum until all fragments are more than radius away,
// since then even if a local minimum gets found it will no longer produce
// the black shading.
while (minDist < radius && minDist >= 0.0 && iter < Math.Max(screenWidth, screenHeight))
minDist = sampleNeighborhood(iter++, worldPos);
if (minDist < 0.0) //found a local minimum
return vec3(1,1,1); //or any other computed shading
//I am aware that the function call above checks central fragments several times.
//That should be of course optimized in production code.
float sampleNeighborhood(int neigh, vec4 worldPos)
float minDist = float.MaxValue;
for(int x = Math.max(0, frag.x - neigh); x <= Math.Min(screenWidth, frag.x + neigh; ++x)
for(int y = Math.max(0, frag.y - neigh); x <= Math.Min(screenHeight, frag.y + neigh; ++y)
float p = tex2D(distance_texture, x, y);
float d = distance(p.xyz, worldPos.xyz);
if (p.a > 0.5 && d <= radius)
//Return negative distance to indicate a local minimum was found.
minDist = Math.min(minDist, d);
Finding the local minima in the second pass is much easier, since you only need to sample the local 3x3 neighborhood.
I hope that helps you in reaching the desired rendering!