Shader code for putting an atmosphere over a planet

Question

I'm working on a game set in space with procedurally generated planets, and I'm trying to write a shader that puts an atmosphere over a planet given the color of the atmosphere and the strength of the atmosphere.

It looks ok at low strengths:

But at high strengths the planet looks really bright:

And neither of them look realistic. For example in space engine the atmosphere really affects every part of the planet including the oceans without necessarily looking very bright:

Here's the relevant part of the shader code I'm using (in glsl):

//Final atmosphere color to use
vec4 atmo_pixel_color;

if(r > radius_percent)
{
    //This is in space, cut off the color quickly
    float atmo_alpha = clamp(0.5 - pow((r - radius_percent), 2) * 600, 0, 1);
    atmo_pixel_color = atmo_color * atmo_strength;
    atmo_pixel_color.w = atmo_alpha;
}
else
{
    //Atmosphere is thicker near the edges of the planet
    float thickness = pow((r / radius_percent), 2) * 0.1f;
    atmo_pixel_color = atmo_color * (atmo_strength + thickness);
}

gl_FragColor = planet_color + atmo_pixel_color;

r is the distance from the center of the planet. radius_percent is the percentage of the texture that is in the planet (above that is empty space). planet_color is the final base color of the planet itself, or empty black if the pixel is in space.

I have very little experience doing graphics programming and I didn't really know where to start to research this.

What kind of algorithm should I be using here to make the atmosphere look correct?

Answer 1

I believe that Space Engine as a base algorithm still uses Precomputed Atmospheric Scattering by E. Bruneton. It is released on github. So it is good starting point if you want to implement atmospheric scattering.

For beginner this algorithm is pretty advanced stuff written in C++11, and modern OpenGL, so such knowledge is required. In short, to avoid a cost of solving rendering equation for scattering in participating medium (single as well as multiple scattering) it uses precomputed lookup textures where it is possible and equations for Rayleigh and Mie scattering based on physical model.

There is also simpler demo computed in real time on shadertoy. This is probably better place to start. It involves raycasting a sphere and integrating Rayleigh and Mie scattering along a ray in real time. It is based on GPU Gems article, which provides some good information how it works.