Path tracer not rendering shadows

Question

I am trying to implement for research purposes a gradient-domain path tracer. In order to achieve this step I first need a working Path tracer. I have been creating one so far but results are wrong and I will explain you why. Some concepts:

I am working on paths that are generated before sampling them. I mean that the first step of my algorithm consists in calculating a path for a certain pixel (x,y). This path will perform some bounces within the scene and, if terminates on the light, will be considered valid.

First of all: the following is the definition of a struct (will be declared and initialized later), containing some scene info needed to cast ray from the camera into the scene.

struct RenderData
{
    vec3 E;
    vec3 p1;
    vec3 dx;
    vec3 dy;
};

It is initialized in the InitializeScene method:

void InitializeScene(){ // setup virtual screen plane
  vec3 E( 2, 8, -26 ); //Eye position
  vec3 V( 0, 0, 1 ); //LookAt vector
  float d = 0.5f, ratio = SCRWIDTH / SCRHEIGHT, focal = 18.0f;
  vec3 p1(E + V * focal + vec3(-d * ratio * focal, d * focal, 0)); // top-left screen corner in SCREEN SPACE
  vec3 p2(E + V * focal + vec3(d * ratio * focal, d * focal, 0)); // top-right screen corner
  vec3 p3(E + V * focal + vec3(-d * ratio * focal, -d * focal, 0)); // bottom-left screen corner
  mat4 M = rotate( mat4( 1 ), r, vec3( 0, 1, 0 ) );
  p1 = vec3(M * vec4(p1, 1.0f)); //rotating the above points
  p2 = vec3(M * vec4(p2, 1.0f));
  p3 = vec3(M * vec4(p3, 1.0f));
  renderData.dx = (p2 - p1) / (float)SCRWIDTH;
  renderData.dy = (p3 - p1) / (float)SCRHEIGHT;
  renderData.E = vec3(M * vec4(E, 1.0f));
  renderData.p1 = p1;
}

The code above is to make you understand how I initialized the scene. I also have structs to store informations about my paths:

struct PathVert {
   vec3 p; vec3 n; //hit point and normal of the surface hit 
};

struct Path {
   PathVert verts[MAX_DEPTH]; //maxDepth is 15 for now
   int vertCount;
   int x, y; //which pixel this path is referring to
};

therefore I start consider pixel after pixel.

for (int y = 0; y < SCRHEIGHT; y++) for (int x = 0; x < SCRWIDTH; x++)
{     
   Path path;
   if(generatePath(x,y, path)){
      Sample(path);
   }
}

The generatePath() method indeed tracks the path into the scene and checks all the vertices it hits. The checkIfRayIntersectSomething(t) method you will see used, it's just a pseudo method implemented in my framework and that I omit posting cause of its length. I use it to check if my ray hits something in the scene, if it does, it update the "t" with the distance to that object. NOTE: the light is not considered an object itself. Hence, I also have a checkRayLightIntersection(hitLightPoint) which checks the intersection with the light, if there is any, the hitLightPoint is updated with the point on the light I have been hitting. The light is a 2D surface.

Vec lightPos = Vec(5, 15, 2); //hard coded position of the light

And the light, as said, is a surface, but exactly a square surface, whose 4 angles are:

Vec P1 = Vec(lightPos.x - 20, lightPos.y, lightPos.z + 20);
Vec P2 = Vec(lightPos.x + 20, lightPos.y, lightPos.z + 20);
Vec P3 = Vec(lightPos.x + 20, lightPos.y, lightPos.z - 20);
Vec P4 = Vec(lightPos.x - 20, lightPos.y, lightPos.z - 20);

Quite big, I know, so the first question relies on this aspect, is it correct having such a big light?

But let's go to the main methods. Hereby you can see the GeneratePath method:

bool GeneratePath(int x, int y, Path &path){
   path.x = x;
   path.y = y;
   path.vertCount = 0;

   vec3 P = renderData.p1 + renderData.dx * ((float)(x) + Rand(1)) +  renderData.dy * ((float)(y) + Rand(1));
   vec3 O = renderData.E + vec3(Rand(0.4f) - 0.2f, Rand(0.4f) - 0.2f, Rand(0.4f) - 0.2f);
   vec3 D = normalize(P - O); //direction of the first ray, the one from the camera towards the pixel we are considering

   for (int depth = 1; depth <= MAXDEPTH; depth++){
    float t;
    Vec hitLightPoint;
    PathVert vert;
    if (!checkIfRayIntersectSomething(t)){
        //we didn't find any object.. but we still may have found the light which is an object non represented in the scene
        //the depth check avoids me rendering the light as a white plane
        if (depth > 1 && checkRayLightIntersection(O, D, hitLightPoint)){
            //update the vertex since we realized it's the light
            vert.p = hitLightPoint;
            vert.n = Vec(0, -1, 0);//cause the light is pointing down
            path.verts[depth - 1] = vert;
            path.vertCount++;
            return true; //light hit, path completed    
        }
        return false; //nothing hit, path non valid
    }
    //otherwise I got a hit into the scene
    vert.p = O + D * t; //reach the hitPoint
    vert.n = methodToFindTheNormal();
    vert.color = CalculateColor(vert.p); //according to the material properties (only diffuse objects so far)
    path.verts[depth - 1] = vert;
    path.vertCount++;

    //since I have the light, and a path terminates when it hits the light, I have to check out also if my ray hits this light,
    //and if does, I have to check whether it first hits the light or the object just calculated above
    //moreover with the "depth > 1" check, I avoid again rendering the light which otherwise would be visible as a white plane

    if (depth > 1 && checkRayLightIntersection(O, D, hitLightPoint)){
        float distFromObj = length(vert.p);
        float distFromLight = length(hitLightPoint);
        if (distFromLight < distFromObj){
            //update the vertex since we realized it's the light
            vert.p = hitLightPoint;
            vert.n = Vec(0, -1, 0);
            vert.color = Vec(1, 1, 1);// TODO light color? or light emission?

            path.verts[depth - 1] = vert;
            return true; //light hit, path completed
        }
    }
    if (depth == MAXDEPTH) return false;
       Vec newDir = BSDFDiffuseReflectionCosineWeighted(vert.n, D);//explained later
       D = newDir;
       O = vert.p;
   }
   return false;
}

The BSDFDiffuseReflectionCosineWeighted() just calculate the new directions, tested and working. What remains last is the Sample method which calculates the final color of the pixel.

Vec Sampling(Path &path){

   Vec color(1, 1, 1);

   for (int vert = 0; vert < path.vertCount - 1; vert++) { //considers the last vertex as the light
      const PathVert &currVert = path.verts[vert];
      const PathVert &nextVert = path.verts[vert + 1];
      Vec wo = (nextVert.p - currVert.p).norm();
      double cosTheta = fabs(wo.dot(currVert.n));
      float PDF = cosTheta/PI;
      if (cosTheta <= 1e-6) return Vec();
      //considering only DIFFUSE objects
      color = color.mult(currVert.color * (cosTheta / M_PI) / PDF);
   }
   return color.mult(Vec(10.0f, 10.0f, 10.0f)); //multiplication for the light emission?
}

Result with 16SPP is:

As you can see the results are not that bad but there is one main problem: shadows are missing.. tried many combinations but no improvements. There is some error in the algorithm itself. Can you help me understanding why? Thanks in advance.

EDIT: this is my target reference:

EDIT 2: this is what happens in case the depth > 1 check is omitted:

EDIT 3: after fixing the distFromLight and distFromObj I got the following results:

One of the 2 problems as suggested by @trichoplax could be too soft shadows. So I tried to lower the size of the light, increasing his power to compensate. Results are not good and shown here:

Moreover as requested I can post some more code:

bool checkRayLightIntersection(Vec O, Vec D, Vec & hitLightPoint){
  //getting the 4 corners of my light
  Vec P1 = Vec(lightPos.x - 20, lightPos.y, lightPos.z + 10);
  Vec P2 = Vec(lightPos.x + 20, lightPos.y, lightPos.z + 10);
  Vec P3 = Vec(lightPos.x + 20, lightPos.y, lightPos.z - 10);
  Vec P4 = Vec(lightPos.x - 20, lightPos.y, lightPos.z - 10);

  //the majority of the methods first find out where the ray intersects the plane that the rectangle lies on Ax + By + Cz + D = 0
  //in our case the equation of that plane is easy -> D = 20
  // answers.google.com/answers/threadview?id=18979

  float t = -(-O.y + lightPos.y) / (-D.y);
  if (t > 0){
    Vec hitPoint = O + D * t;
    Vec V1 = (P2 - P1).norm();
    Vec V2 = (P3 - P2).norm();
    Vec V3 = (P4 - P3).norm();
    Vec V4 = (P1 - P4).norm();
    Vec V5 = (hitPoint - P1).norm();
    Vec V6 = (hitPoint - P2).norm();
    Vec V7 = (hitPoint - P3).norm();
    Vec V8 = (hitPoint - P4).norm();

    if (V1.dot(V5) > 0.0 && V2.dot(V6) > 0.0 && V3.dot(V7) > 0.0 && V4.dot(V8) > 0.0){
        hitLightPoint = hitPoint;
        return true;
    }
  }
  return false;
}

Instead, the checkIfRayIntersectSomething method is part of the framework itself (it has just another name), it's not done by me but it's tested and working.

EDIT 4: tried to render a small part of the scene with high number of spp. Tried 512 spp for the bottom right corner where should appear shadows according to the target image. Results show that shadows are not there:

EDIT 5:

I hereby post the code used to render the target image, it does at the same time my Sampling() and GeneratePath() methods using a recursive structure.

Vec Sample(vec3 O, vec3 D, int depth)
{
  vec3 color(0, 0, 0);
  float t;
  if (checkIfIntersectSomething(D, t))
  {

    vec3 I = O + t * D; //get to the intersection point on the object
    vec3 diffuse = getMaterial(I);

    vec3 L = vec3(-1 + Rand(2.0f), 20, 9 + Rand(2.0f)) - I; //(-1,20,9) is Hard-code of the light position, and I add Rand(2.0f) on X and Z axis
    //so that I have an area light instead of a point light
    L = normalize(L);
    float ndotl = dot(I.getNormal(), L); //the closer the dotProdutc is to 1.0, the more the light and surface face each other
    if (ndotl > 0)
    {
        if (!checkRayLightIntersection(L)) {
            float dist = distFromLight(I);
            color += diffuse * ndotl * vec3(1000.0f, 1000.0f, 850.0f) * (1.0f / (dist * dist));
        }
    }

    // russian roulette
    float Psurvival = CLAMP((diffuse.r + diffuse.g + diffuse.b) * 0.33333f, 0.2f, 0.8f);

    // continue random walk 
    float rand = Rand(1.0f);
    if (depth < 10 && rand < Psurvival)
    {
        //Besides russian roulette, I also do another weight, because rays that go towards the horizon will bring back very little energy
        //so I make a random distribution that favours those rays who are close to the normal of the hit point, this is DiffuseReflectionCosineWeighted(). It creates a Random bounce but proportional to N dot R
        vec3 R = DiffuseReflectionCosineWeighted(I.getNormal());//there is a weight

        float prob = 1.0;
        float cosTheta = fabs(dot(I.getNormal(), R));
        if (cosTheta > 1e-6) prob = cosTheta / M_PI;

        color += diffuse * Sample(I + R * EPSILON, R, depth + 1) * (1.0f / Psurvival); //the cosTheta() of the attenuation of the rendering equation gets simplified with the cosTheta of the "prob"
        //the PI of the prob gets simplified with the BRDF where we are using the ideal BRDF = diffuse/PI
    }
}
return color;

}

EDIT 6: Trichoplax was right, I had a problem calculating the distanceFromObject and distanceFromLight. After solving this, we realized that some sort of shadows appeared (shown in one of the images above). He then suggested that such a broad light might have been washing out the shadows, and so it was. By reducing the light size and by using a high amount of SPP we realized that shadows are there and visibile: https://i.stack.imgur.com/Eez7y.jpg . As a drawback tho, way less rays reach the light, giving back an image with high noise, but this is another kind of problem and I created another question for it. Thanks to @trichoplax and @julien for their help

Answer 1

The problem appears to be unintentionally transparent surfaces

Although the image is grainy, it is sufficiently clear to estimate that all of the darker regions are due to surfaces facing away from the light, rather than due to shadows cast onto surfaces facing towards the light. So it does seem that there is a problem, and the lack of shadows is not just due to having soft shadows from a large area light that don't show up well, or excessively reflective surfaces that provide too much ambient light. Somehow light is getting through your surfaces.

The probable source of this problem

Since you check for intersection with the light and the objects separately, the crucial point where this could go wrong is where you check

if (distFromLight < distFromObj)

You correctly terminate the path and return the light colour if this check is true (that is, if the light is closer than the nearest object in that direction). So the problem is likely to be with one of the two variables you are comparing. If they were both correct, then you would see shadows.

These are calculated on the preceding lines

float distFromObj = length(vert.p);
float distFromLight = length(hitLightPoint);

As pointed out in Julien Guertault's answer, these calculations may be redundant.

length(vert.p) should still be available as t, used 11 lines previously in the calculation of vert.p. This not only avoids unnecessary calculation, but eliminates a potential source of error. Then this length is likely to be correct unless there is a problem with checkIfRayIntersectSomething().

This leaves only length(hitLightPoint) as a source of error, unless there is a more subtle problem. It would be useful to be able to eliminate the possibility of problems with length() or hitLightPoint, which is calculated by checkRayLightIntersection().

We need to see the code to be able to help

length(), checkIfRayIntersectSomething() and checkRayLightIntersection() are not currently included with the code in your question, so we can't analyse these. If you are able to share these that would help narrow down the source of the problem.

A simple way to check whether this is the real problem

You could also verify whether this analysis of the problem is correct by rendering a scene which will eliminate the other possibilities (shadows too soft to distinguish, or shadows washed out by too much reflected light from the rest of the scene). If the light getting through the surfaces is the source of your problem, then it will apply even if there is no path from the light to a surface, provided that surface is facing the light. So render a simple scene that has a light with a plane beneath it, and another plane beneath that, separated by a gap big enough to easily be seen. The lower plane is completely occluded by the upper plane, so should receive no light (no soft shadows and no reflected light from elsewhere). If the lower plane is lit, then you know that this is indeed the problem and you can focus your efforts here.

If the lower plane is not lit, then either this is not the problem, or there is a problem with the scale and the occluding plane may need to be at a different distance from the light in order for the problem to show up. You can use your existing scene as a source of estimated distances for which this problem occurs, and you can use inclined planes in order to see the effect for a variety of ratios of object distance to light distance within a single image.

Sharing the images resulting from such simple scenes may help us identify the problem more precisely.

---

I notice that in a comment on the question you mention that you do not have a straightforward way of generating scenes yourself. Perhaps there are much simpler scenes already publicly available in the same file format, that you can use to test this?

If not, it sounds like the only thing you have control over is the light. In that case, you could try putting the light underground. If surfaces are not transparent, the scene should be uniformly black. If surfaces are being lit despite an occluding surface, you may see evidence for it this way that would be hard to discern when there is lots of secondary light bouncing around.

Answer 2

is it correct having such a big light?

I don't see any problem with having a big area light. That said, it also depends on the scale of your scene. If the light is large compared to it, shadows will tend to be more diffuse, like under an overcast sky.

shadows are missing.. [...] Can you help me understanding why?

I haven't found any blatant mistake, but there are two elements that look suspicious to me.

if (depth > 1 && checkRayLightIntersection(O, D, hitLightPoint)) {
    // ...
    return true; //light hit, path completed
}

Although this part seems to be correct as far as I can tell, I would try to make sure there is no bug introducing a bias. For every bounce, it checks if there is a ray hitting the light first, and if so it breaks out of the loop: in case the code is incorrect and it happens to hit the light more often than it should, the shadows will disappear.

Moreover, in the first if:

return false; //nothing hit, path non valid

and the related loop:

for (int y = 0; y < SCRHEIGHT; y++) for (int x = 0; x < SCRWIDTH; x++)
{     
  Path path;
   if(generatePath(x,y, path)){
      Sample(path);
   }
}

Make sure you still take the paths that hit nothing into account when averaging the color of the pixel, otherwise this will also introduce a bias.

As joojaa said in the comment, you should try using a simpler scene: a few spheres or cubes, no texture. You can also reduce the number of bounces to one or two to see only direct lighting. If necessary, maybe even use a constant sky dome first instead of a light area.

---

Other remarks:

struct Path {
   PathVert verts[MAX_DEPTH]; //maxDepth is 15 for now
   int vertCount;
   int x, y; //which pixel this path is referring to
};

Do you need to store x and y?

float distFromObj = length(vert.p);
float distFromLight = length(hitLightPoint);

Don't you already have two t that you can compare?