I'm brand new to OpenGL and graphics programming in general, and I'm trying to figure out what software I need to program a GPU. I saw an NVidia OpenGL driver, and I've also seen reference to the OpenGL SDK. My understanding (sort of) is that the OpenGL driver is essentially an API that hides the differences between different GPUs and the OpenGL API works on top of that. I really have no idea though.
I wouldn't really call it OpenGL SDK but OpenGL API. There's a common OpenGL API which is basically a set of functions you can call to control the GPU. This API is hardware agnostic (barring extensions), so you can use it on NVIDIA, ATI, etc. GPU's.
What the OpenGL driver does is to turn these platform agnostic function calls to hardware specific commands that the driver then sends to the GPU. For example when you tell GPU to draw a triangle via OpenGL API, then the driver converts these calls under the hood to specific commands for the specific GPU you are using. This abstraction is important so that you don't have to write different code for every GPU.
To explain things a bit in lower level, the graphics API implementation operates in user mode, while the graphics driver operates in kernel mode. User mode is "protected" mode your program runs in, in the sense that you can't mess up your entire device but only your own app and it's also relatively fast to perform the API function calls in this mode. Kernel mode on the other hand is "unprotected" and performing kernel mode calls requires an expensive mode switch. For OpenGL the IHV's provide both the OpenGL API and driver implementations and thus IHV's can provide optimal switching between the modes. This is in contrast to DirectX where API implementation is provided by Microsoft and the driver by IHV's.