I have a scene set up with an orthographic projection with the near / far set from -5000 to 5000. The model view is centered at 0 in the z-axis. I am rendering a sphere centered at z=0 with lighting, and the specular component appears as to be calculated as if the eye were infinity far back. See image below, in which the light only has a red specular component, no diffuse or other parameters. However, if i render this with a custom shader that implements blinn lighting, the sphere appears black, because the "viewDir" calculation is negative, since the eye is centered inside the sphere. How does fixed function get around this? Does it some how know about the ortho projection and get around it? Sources I've seen online say lighting is not supposed to take the projection into account.
This isn't necessarily due to the orthographic projection per se, but rather the classic two-sided lighting problem together with a little optimization that the old OpenGL did. When the eye space origin is inside the object, you are basically looking at the object from the inside and it's only the peculiarities of orthographic projection and the near/far planes you chose that makes the outsides appear.
It depends on how you actually implement your shader. I assume though, when you compute your
viewDir, you just use the vector from the eye space vertex to the origin (or rather the negative normalized eye space vertex position). You are basically trying to light the backfaces of the sphere and if you haven't taken account for that, the angle between the view vector and the normal will thus be larger than 90°, usually resulting in a negative or 0-clamped specular term (or whatever other unexpected results).
A common way to accomodate for that would be to negate the normal vector whenever it is facing away from the camera, or maybe the light, or when dealing with backfacing fragments,... But this depends on your specific use-case.
Now, why doesn't this happen in old deprecated OpenGL when using orthographic projection?
For this we have to look at the specificities of how it did its lighting computations. There was a function
glLightModel which controlled some of the global properties of lighting computations. Specifically, the paramter
GL_LIGHT_MODEL_LOCAL_VIEWER. This controls an optimization where the specular computation does not use the actual vector from the vertex to the camera, but always just uses the z-axis instead. i.e. as you said it assumes an infinitely far away viewer along the z-axis. This is the reason why even in perspective projection the specular highlight on an object is always at the same position, even if you move it around in the view. If you change this setting from the default of
GL_TRUE, it should perform an actual view vector computation and you should get the same results as with your own shader.
There was also another option that controls a property related to your general issue, too.
GL_LIGHT_MODEL_TWO_SIDE enables two-sided lighting, which means that back-facing polygons will be lit like their front faces (however, possibly using different material settings) by using the inverted normal. If you activate this, this still won't change anything in your orthographic scene, since you are seeing the front faces of the sphere here. But once you change to perspective projection, the lighting should be black, but this time because you're seeing the backfaces, which you can enable lighting for with this option.