interpolated normal vectors after tessellation. How to hash patch corner / edge vertices?

Question

I have a 3D geometry (an asteroid) that is to be tessellated with respect to the camera position. Each vertex in the original mesh has a unique position and is stored once (VBO with IBO)

In the evaluation phase of the tessellation, I shift the vertices with several layers of Perlin noise, curved PN triangles, and some crater kernels. So that the normal vector for each vertex cannot be computed analytically. Moreover, there is no way to interpolate the vertex parameters of the tessellation evaluation for the next stage. (At least I have not found such an interpolation option)

For this reason, I considered disabling the rasterization and storing the vertices (with position/UV/normal) in an SSBO buffer as a later VBO. The triangle index connections are also stored in a separate SSBO as a later IBO. Similar to transformation feedback.

struct Vertex
{
    vec4 position;
    ivec4 normal; //here ivec4 is used to be able to do atomicAdd(). The normal vector is NOT normalized!
    vec4 additionalInformation; //in here I can store UV coordinates or other information
};

After generating the VBO and IBO, and before rendering these VBO/IBO, I want to add an additional compute shader to compute the normal vectors of each vertex with respect to the adjacent triangles.

To do this, each compute shader-invocation loads a triangle from the IBO and uses a hash function to find the three vertex information (VBO). Compute the three angles and the normal vector of the triangle. Then the normal vector is multiplied by $1000 * angle_{rad}$ to get high numbers to store the normal vector in integer variables (atomicAdd) in the VBO buffer.

Thus, when rendering the created VBO/IBO later, the shader only needs to load the integer (non-normalized) normal vector, store it in a vec3 and normalize it, and the surface can be rendered with interpolated normal vectors.

To do this, several problems have to be solved:

Problems I was able to solve:

• Tessellation evaluation: generation of a unique vertexID per created vertex of a patch (triangle)
• Tessellation evaluation: Generation of the IBO connections

Problems that need to be solved:.

I don't need hashing for vertices that are inside the triangle patch... because the IBO point directly to the VBO indexes. But at the three vertex corners and three edges of each triangle patch, the vertices would be stored multiple times. So normal interpolation would only work for faces that are not edge or corner faces of the original mesh. That is, the resulting image would look smooth, only the edges of those faces would look edgy....

I thought about this for a long time and came to the conclusion that only the edge and corner points need to be hashed. But what should the hash key look like?

I came up with 3 solutions:

I just want to have a 32-bit hash key for performance and atomic function reasons.

1. the 3D position is hashed by taking 10/11/11 bits from each mantissa (x,y,z) and storing them in a 32-bit integer hash value. I think the mantissa is the right choice because the exponent is very often identical when two vertices are close to each other.
2. combine the two patch corner vertex IDs in the following way: 13 bits of the smaller vertex ID + 13 bits of the larger vertex ID + 6 bits of the outer tessellation ID (values: 0 - 63). The problem with this is that with the 13 bits of vertexID, I can only have a input mesh with a maximum of 8192 vertices. Which is way too small :(.
3. do it as in option 2. with the following changes: Take 23 bits of the smaller vertexID + 6 bits of the outer tessellation id (values: 0 - 63) + 3 bits for an edgeID. Thus, each vertex in the original mesh can be connected to a maximum of 8 other vertices (3 bits) to be computed when loading the mesh to the GPU. Therefore, the original mesh can have 8,388,608 vertices, which is still not enough. I don't like this option because calculating the pre-edge connection in the first step and loading this information into the shader, as well as being limited to 8 million vertices and 8 connections, is a significant overhead and restriction.

Do you have a better idea how to hash the patch corners / edges vertices?

Or do you have a better idea how I can access these generated vertices without hashing?