Transforming fragment-to-camera to tangent space for parallax occlusion mapping

Question

I'm trying to implement POM using this tutorial. As stated, I need to transform fragment-to-camera to tangent space and then pass it to a function.

• tangent space is computed using Assimp. Well I trusted it. For example tangent vectors look like that (Sponza model):

• then in vertex I make transformation as in tutorial:

  layout (location = 0) in vec3 vPosition;
  layout (location = 1) in vec2 vTexCoord;
  layout (location = 2) in vec3 vNormal;
  layout (location = 3) in vec3 vTangent;
  layout (location = 4) in vec3 vBitangent;
  ...
  worldPos = matrices.modelMatrix * vec4(vPosition, 1.0);
  //transform t, b, n from object to world (also called model) space?
  vec3 t = normalize(mat3(matrices.modelMatrix) * vTangent);
  vec3 b = normalize(mat3(matrices.modelMatrix) * vBitangent);
  vec3 n = normalize(mat3(matrices.modelMatrix) * vNormal);
  mTBN = transpose(mat3(t,b,n)); //transpose makes it from model to to tangent space?
  tanViewPos = mTBN * cameraPos;
  tanFragPos = mTBN * worldPos.xyz;

• then in fragment I pass parameters to function:

  vec3 viewDir = normalize(tanViewPos - tanFragPos);
      texCoord = steepParallaxMapping(vTexCoord0, viewDir, textureHeight);

  vec2 steepParallaxMapping(vec2 texCoords, vec3 viewDir, sampler2D textureHeight)
  {
  float height_scale = 0.09;
  // number of depth layers
  const float numLayers = 30.0;
  // calculate the size of each layer
  float layerDepth = 1.0 / numLayers;
  // depth of current layer
  float currentLayerDepth = 0.0;
  // the amount to shift the texture coordinates per layer (from vector P)
  vec2 P = viewDir.xy * height_scale;
  vec2 deltaTexCoords = P / numLayers;
  // get initial values
  vec2  currentTexCoords     = texCoords;
  float currentDepthMapValue = texture2D(textureHeight, currentTexCoords).r;
  
  
  while(currentLayerDepth < currentDepthMapValue)
  {
      // shift texture coordinates along direction of P
      currentTexCoords -= deltaTexCoords;
      // get depthmap value at current texture coordinates
      currentDepthMapValue = texture2D(textureHeight, currentTexCoords).r;
      // get depth of next layer
      currentLayerDepth += layerDepth;
  }
  
  // get texture coordinates before collision (reverse operations)
  vec2 prevTexCoords = currentTexCoords + deltaTexCoords;
  
  // get depth after and before collision for linear interpolation
  float afterDepth  = currentDepthMapValue - currentLayerDepth;
  float beforeDepth = texture2D(textureHeight, prevTexCoords).r - currentLayerDepth + layerDepth;
  
  // interpolation of texture coordinates
  float weight = afterDepth / (afterDepth - beforeDepth);
  return prevTexCoords * weight + currentTexCoords * (1.0 - weight);
  }

  Thats the result (simple displacement):

Or other texture (complex):

I'm doing something wrong. And I wonder why we multiply by 3x3 part of the model matrix?

EDIT:

I've found function which calculates tbn in fragment:

mat3 compute_tangent_frame(vec3 N, vec3 P, vec2 UV)
{
    // get edge vectors of the pixel triangle
    vec3 dp1 = dFdx( P );
    vec3 dp2 = dFdy( P );
    vec2 duv1 = dFdx( UV );
    vec2 duv2 = dFdy( UV );

    // solve the linear system
    vec3 dp2perp = cross( dp2, N );
    vec3 dp1perp = cross( N, dp1 );
    vec3 T = dp2perp * duv1.x + dp1perp * duv2.x;
    vec3 B = dp2perp * duv1.y + dp1perp * duv2.y;

    // construct a scale-invariant frame
    float invmax = inversesqrt( max( dot(T,T), dot(B,B) ) );
    return mat3( T * invmax, B * invmax, N );
}

First Case: So if I use that function like that:

//in vertex
vFar = cameraPos - worldPos.xyz; 
//in fragment
mat3 TBN = (compute_tangent_frame(vNormal0, normalize(vFar), vTexCoord0));
vec3 viewDir = (transpose(TBN)*normalize(cameraPos - worldPos.xyz));
vec2 texCoord = steepParallaxMapping(vTexCoord0, viewDir, textureHeight);
vNormalExtr = normalize(texture2D(textureNormal, texCoord).rgb*2.0 - 1.0);
normalMapColor = vec4(normalize((TBN) * vNormalExtr), 1.0);

parallax (all walls) and normal light seem to be ok:

Second Case: But if I use my TBN from CPU:

//in vertex
mat4 model = ((matrices.modelMatrix));
vec3 t = normalize(mat3(model) * (vTangent));
vec3 b = normalize(mat3(model) * (vBitangent));
vec3 n = normalize(mat3(model) * (vNormal));
mTBN = ((mat3(t,b,n)));
//in fragment
mat3 TBN = mTBN;
vec3 viewDir = (transpose(TBN)*normalize(cameraPos - worldPos.xyz));
vec2 texCoord = steepParallaxMapping(vTexCoord0, viewDir, textureHeight);
vNormalExtr = normalize(texture2D(textureNormal, texCoord).rgb*2.0 - 1.0);
normalMapColor = vec4(normalize((TBN) * vNormalExtr), 1.0);

parallax is still twisted on all walls but normal light seem to be ok:

If I dont transpose tbn (viewDir) (the correct way You said) in second case (mTBN)

vec3 viewDir = ((TBN)*normalize(cameraPos - worldPos.xyz));

then some walls seem to be correct, others not...

EDIT:

Adventures in tangent space... Is there a badge of longest question ever or biggest number of pictures?

Answer 1

Without looking at the code, from the look of the pictures, my intuition would be that somewhere the code is using the inverse of the matrix it should.

Looking at the code seems to confirm so, but there are some other problems too.

mTBN = transpose(mat3(t,b,n)); //transpose makes it from model to to tangent space?

mat3(t,b,n) is a transform from world space to tangent space. The inverse of such a matrix is indeed its transpose, so mTBN transforms from tangent space to world space, which is not what you want.

tanViewPos = mTBN * cameraPos;
tanFragPos = mTBN * worldPos.xyz;

You cannot do something like that. mTBN is a 3x3 matrix: without a 4th column and a non-zero homogeneous coordinate, you can only transform directions, not positions. You need to first get the camera direction relative to the vertex, then transform that direction.

// Beware of the vector direction.
// Here viewDir points from the object toward the viewer.
vec3 worldViewDir = normalize(cameraPos - worldPos.xyz);
tanViewDir = mTBN * worldViewDir;

Let us know in comments if that helps solving your issue.

Answer 2

you don't actually need to transform the z component, just ommit it and replace it with 1,because it is in tangent space it will only cause problems for this edit: I had the same problem and that fixed it