This question is NOT about the hardware difficulties and shortcomings of tesselation shaders
I am trying to understand the motivation behind tesselation shaders.
Why/when/where tesselation is a good idea?
Wouldn't the new triangles formed after tesselation lie on the same plane as the original triangle, thus have the same normal?
How is that going to help the final drawing being more realistic?
Wouldn't the new triangles formed after tesselation lie on the same plane as the original triangle, thus have the same normal?
No. The domain shader / tessellation evaluation shader can move the new vertices anywhere you like based on shader logic, so the generated surface need not be flat.
Tessellation can be used to evaluate curved surfaces such as bicubic patches of various kinds, or apply displacement mapping. See Phong tessellation for an example of this.
Sometimes, there are also reasons to use tessellation even if you place the generated vertices on the same plane. For example, you can evaluate lighting terms in the domain shader, enabling more detailed lighting than if you did it in the vertex shader, but at less expense than if you did it in the pixel shader. This can be useful for particles, as seen in the presentation Practical Particle Lighting from GDC 2012.
