In OpenGL, there are 6 different shader types:
- Vertex Shader
- Tesselation Control Shader
- Tesselation Evaluation Shader
- Geometry Shader
- Fragment Shader (Pixel)
- Compute Shader
The first 5 are part of the render pipeline and fulfill a very specific purpose in that pipeline. They are executed in ascending order. The shader types 2.-4. are optional and can be skipped. Between those stages, some operations are performed, that are very specific to rendering. Have a look at this link for further information.
The problem here is: Since the rendering pipeline has a very specific purpose (generating an image of pixels), it is very hard to use it and its shaders for something else than rendering. Won't say it is impossible, but it gets certainly harder and harder the less your goal has something to do with pixel generation.
For general-purpose computations, you can use compute shaders. You can basically compute anything there, so this is probably the shader type you are looking for. However, the way you ask your questions gives me the feeling, that you have some misunderstandings on how rendering works in general and you should probably read some literature about how the render pipeline creates an image from 3d data.
But let’s say you had a shader that created a silhouette (outline) around some geometry. Is it possible for a shader program to generate vector graphics (ie. Curves) that follow that exact silhouette (instead of pixel data)?
As written above, generally yes. It is possible to write a shader that does that. But probably not with the same technique as the outline is created in the "pixel case". It is certainly much more complicated.
Or in a different example, this image was created by using a pixel shader. Would it be possible for a shader to generate the lines on a teapot as actual curve geometry rather than just a bitmap that is mapped to the 3D geometry?
Again, yes it is possible for a shader to fulfill this task, but you wouldn't use a pixel (fragment) shader for that. You would also need to approach the whole task very differently.
Are shader programs fundamentally different and as such could not generate lines, curves, etc. as a form of output?
The shaders of the rendering pipeline are tied to a very specific task and have very specific boundaries on their inputs and outputs. But if you know what you are doing, you can probably hack around those limitations. However, I think it wouldn't make much sense to do so, except you want to prove that you can.
Instead, you can use a compute shader for any computation, that has nothing to do with rendering.
To sum it up
You can use compute shaders to get other output types than just pixels and images made of pixels. However, calculating basic 2d shapes/curves/vector graphics from 3d data will most likly be fundamentally different than generating a corresponding bunch of pixels from that same data. In conclusion, you can't just turn a switch (change 2-3 lines of code) to select which output you like to have.