Raytracing is a great way to render spheres as Daniel mentioned. Below is a screenshot of a raytraced scene I made but lost the source code to unfortunately.
If you are looking to render smooth / non polygonal objects in general, it's not always easy, desired or even possible to do an analytical solve for ray vs object.
For these cases, a useful technique is something called "ray marching" (also called sphere tracing). In ray marching, you take steps down a ray and at each point ask if you are inside or outside of the object (alternately, above or below). When the answer changes, you know that the intersection of the ray occurred between the current and last point you tested on the ray.
Once you know the intersection occured between those two points, you can do various things to try and find the actual intersection, depending on quality needs vs computation costs. Some common techniques:
- Just accept either the current or last point tested as the intersection point (fastest)
- Binary search the range to get a better answer
- If you have a function which gives you an (estimated) signed distance from a point to the surface, instead of just an "inside vs outside" test, you can do a linear interpolation between the last point and the current point to find where the "0" is, and take that as your intersection point. This is what I've had most success with personally.
Note that you can get an estimated signed distance numerically using finite differences and using the gradient. Check this link for more info on that: http://www.iquilezles.org/www/articles/distance/distance.htm
You can find more info about finite difference methods here:
With a signed distance estimating function, you can also use the gradient to get the normal, which is required for shading.
Below is a ray marched surface that is defined as a function $z=f(x,y)$, and is a screenshot of this webgl2 interactive demo: http://demofox.org/LeastSquaresSurfaceFit.html
Alternately, here's a shadertoy which ray marches a cubic bezier rectangle - it's cubic on each axis, so is degree(3,3): https://www.shadertoy.com/view/4tfXz2
Back to talking about rendering spheres, you can also just rasterize a sphere as a 2d circle (possibly distorted by a view / projection matrix type setup in "2d"), and then shade it as if it was a 3d object. It can also work for non spheres, and you see this a lot with renderings of metaballs.
Below is a screenshot of this shadertoy which renders metaballs: https://www.shadertoy.com/view/4sXXWM
Lastly, I wanted to mention that all of these techniques are compatible with traditional renderers as well. Like Daniel said, you can draw a screen space billboard that is larger than the shape (or draw a world space scaled and oriented unit cube that contains it) and then you do your raytracing, raymarching, or rasterization / fake 3d lighting within the pixel shader, to mix these techniques with standard rasterization rendering methods.