Parallax/relief mapping with gl_FragDepth and conservative depth

Parallax mapping in any of its ray-marchy variants (e.g. steep parallax, cone step mapping, POM) generates fake geometry on a material, generally nearly indistinguishable from real geometry and at a much higher resolution than is practical using GPU tessellation. However, the geometry created does not extend outside the initial bounds of the triangle being rendered (unlike with GPU tessellation) - so e.g. a parallax mapped sphere still has circular edges.

I'm want to try adapting a parallax shader (preferably OpenGL, any high quality variation of parallax mapping will do though I gather cone stepping is pretty much the state of the art) to write outside of the poly bounds using gl_FragDepth (and enabling conservative depth output).

This seems to be theoretically possible (see this Unity example Modifying depth when parallax mapping | Unity Community). However that post did not include any shader code and got no responses.

Is anyone aware of an existing implementation of this (with source available), or else could give me a good idea of how I should proceed with implementing it?

Many thanks.

Unfortunately, gl_FragDepth won't enable you to write outside of the rasterized triangle's screen-space bounds. All it enables is to modify the values written to the depth buffer within the triangle, replacing the triangle's true depth with a shader-calculated value.

What you're seeing in the screenshot in the Unity forum thread you linked to is two intersecting parallax-mapped surfaces, where the depth output enables occlusion between the two surfaces to be correctly resolved in the screen-space region they both cover. But if you look at the right side of the screenshot, you can see the straight-line edge of the bottom brick polygon.

However! You might still be able to accomplish the goal of getting correct silhouette edges by using discard in the fragment shader when the parallax ray gets outside the bounds of the original polygon. This would involve modifying the ray-march code to check if the march has crossed a triangle edge, and exit with discard if so, rather than continuing to march.

You'd have to find a way to get the information about triangle edges into the fragment shader, though, as this isn't provided by OpenGL. Maybe make your vertex/index buffers available via SSBOs; then you can use gl_PrimitiveID in the fragment shader to index into them and lookup the vertex data.

Unfortunately I don't know offhand of any implementation of something like this, although I'm sure someone out there must have tried it before.

• Thanks. Didn't know about discard - though after thinking about the problem, what I had in mind was returning a flag indicating that alpha should be set to 0 for "out of bounds", which I guess is a clumsy version of discard. I think this only works if the "height map" is interpreted as a "depth map", i.e. we always trace from the eye to points below the real x-y plane of the texture ... which I guess is fine! Can't try it yet, laptop kaput and awaiting new one. Apr 13, 2018 at 22:25
• To clarify, you don't trace from the eye to below the x-y plane of the surface, you still trace to the x-y plane. But when determining whether a point on surface intercepts the ray, all points are below the surface. Thus some rays won't intercept the surface at all. In these cases, we discard/"transparify" them, thus creating a proper silhouette at the edges. Apr 14, 2018 at 12:27
• Problems will still arise at the texture boundary with an adjacent texture whose height map doesn't tile with the current height map - you would see straight through the model at certain angles. Though thinking about it, this would even be the case with "true" geometry from hardware tessellation of a height map - the problem is inherent to inferring geometry from height maps - I'm sure it is surmountable with some thought (and some work in the vertex shader). Apr 14, 2018 at 12:28