I am trying to implement for research purposes a gradient-domain path tracer. In order to achieve this step I first need a working Path tracer. I have been creating one so far but results are wrong and I will explain you why. Some concepts:

I am working on paths that are generated before sampling them. I mean that the first step of my algorithm consists in calculating a path for a certain pixel (x,y). This path will perform some bounces within the scene and, if terminates on the light, will be considered valid.

First of all: the following is the definition of a struct (will be declared and initialized later), containing some scene info needed to cast ray from the camera into the scene.

struct RenderData
{
vec3 E;
vec3 p1;
vec3 dx;
vec3 dy;
};


It is initialized in the InitializeScene method:

void InitializeScene(){ // setup virtual screen plane
vec3 E( 2, 8, -26 ); //Eye position
vec3 V( 0, 0, 1 ); //LookAt vector
float d = 0.5f, ratio = SCRWIDTH / SCRHEIGHT, focal = 18.0f;
vec3 p1(E + V * focal + vec3(-d * ratio * focal, d * focal, 0)); // top-left screen corner in SCREEN SPACE
vec3 p2(E + V * focal + vec3(d * ratio * focal, d * focal, 0)); // top-right screen corner
vec3 p3(E + V * focal + vec3(-d * ratio * focal, -d * focal, 0)); // bottom-left screen corner
mat4 M = rotate( mat4( 1 ), r, vec3( 0, 1, 0 ) );
p1 = vec3(M * vec4(p1, 1.0f)); //rotating the above points
p2 = vec3(M * vec4(p2, 1.0f));
p3 = vec3(M * vec4(p3, 1.0f));
renderData.dx = (p2 - p1) / (float)SCRWIDTH;
renderData.dy = (p3 - p1) / (float)SCRHEIGHT;
renderData.E = vec3(M * vec4(E, 1.0f));
renderData.p1 = p1;
}


The code above is to make you understand how I initialized the scene. I also have structs to store informations about my paths:

struct PathVert {
vec3 p; vec3 n; //hit point and normal of the surface hit
};

struct Path {
PathVert verts[MAX_DEPTH]; //maxDepth is 15 for now
int vertCount;
int x, y; //which pixel this path is referring to
};


therefore I start consider pixel after pixel.

for (int y = 0; y < SCRHEIGHT; y++) for (int x = 0; x < SCRWIDTH; x++)
{
Path path;
if(generatePath(x,y, path)){
Sample(path);
}
}


The generatePath() method indeed tracks the path into the scene and checks all the vertices it hits. The checkIfRayIntersectSomething(t) method you will see used, it's just a pseudo method implemented in my framework and that I omit posting cause of its length. I use it to check if my ray hits something in the scene, if it does, it update the "t" with the distance to that object. NOTE: the light is not considered an object itself. Hence, I also have a checkRayLightIntersection(hitLightPoint) which checks the intersection with the light, if there is any, the hitLightPoint is updated with the point on the light I have been hitting. The light is a 2D surface.

Vec lightPos = Vec(5, 15, 2); //hard coded position of the light


And the light, as said, is a surface, but exactly a square surface, whose 4 angles are:

Vec P1 = Vec(lightPos.x - 20, lightPos.y, lightPos.z + 20);
Vec P2 = Vec(lightPos.x + 20, lightPos.y, lightPos.z + 20);
Vec P3 = Vec(lightPos.x + 20, lightPos.y, lightPos.z - 20);
Vec P4 = Vec(lightPos.x - 20, lightPos.y, lightPos.z - 20);


Quite big, I know, so the first question relies on this aspect, is it correct having such a big light?

But let's go to the main methods. Hereby you can see the GeneratePath method:

bool GeneratePath(int x, int y, Path &path){
path.x = x;
path.y = y;
path.vertCount = 0;

vec3 P = renderData.p1 + renderData.dx * ((float)(x) + Rand(1)) +  renderData.dy * ((float)(y) + Rand(1));
vec3 O = renderData.E + vec3(Rand(0.4f) - 0.2f, Rand(0.4f) - 0.2f, Rand(0.4f) - 0.2f);
vec3 D = normalize(P - O); //direction of the first ray, the one from the camera towards the pixel we are considering

for (int depth = 1; depth <= MAXDEPTH; depth++){
float t;
Vec hitLightPoint;
PathVert vert;
if (!checkIfRayIntersectSomething(t)){
//we didn't find any object.. but we still may have found the light which is an object non represented in the scene
//the depth check avoids me rendering the light as a white plane
if (depth > 1 && checkRayLightIntersection(O, D, hitLightPoint)){
//update the vertex since we realized it's the light
vert.p = hitLightPoint;
vert.n = Vec(0, -1, 0);//cause the light is pointing down
path.verts[depth - 1] = vert;
path.vertCount++;
return true; //light hit, path completed
}
return false; //nothing hit, path non valid
}
//otherwise I got a hit into the scene
vert.p = O + D * t; //reach the hitPoint
vert.n = methodToFindTheNormal();
vert.color = CalculateColor(vert.p); //according to the material properties (only diffuse objects so far)
path.verts[depth - 1] = vert;
path.vertCount++;

//since I have the light, and a path terminates when it hits the light, I have to check out also if my ray hits this light,
//and if does, I have to check whether it first hits the light or the object just calculated above
//moreover with the "depth > 1" check, I avoid again rendering the light which otherwise would be visible as a white plane

if (depth > 1 && checkRayLightIntersection(O, D, hitLightPoint)){
float distFromObj = length(vert.p);
float distFromLight = length(hitLightPoint);
if (distFromLight < distFromObj){
//update the vertex since we realized it's the light
vert.p = hitLightPoint;
vert.n = Vec(0, -1, 0);
vert.color = Vec(1, 1, 1);// TODO light color? or light emission?

path.verts[depth - 1] = vert;
return true; //light hit, path completed
}
}
if (depth == MAXDEPTH) return false;
Vec newDir = BSDFDiffuseReflectionCosineWeighted(vert.n, D);//explained later
D = newDir;
O = vert.p;
}
return false;
}


The BSDFDiffuseReflectionCosineWeighted() just calculate the new directions, tested and working. What remains last is the Sample method which calculates the final color of the pixel.

Vec Sampling(Path &path){

Vec color(1, 1, 1);

for (int vert = 0; vert < path.vertCount - 1; vert++) { //considers the last vertex as the light
const PathVert &currVert = path.verts[vert];
const PathVert &nextVert = path.verts[vert + 1];
Vec wo = (nextVert.p - currVert.p).norm();
double cosTheta = fabs(wo.dot(currVert.n));
float PDF = cosTheta/PI;
if (cosTheta <= 1e-6) return Vec();
//considering only DIFFUSE objects
color = color.mult(currVert.color * (cosTheta / M_PI) / PDF);
}
return color.mult(Vec(10.0f, 10.0f, 10.0f)); //multiplication for the light emission?
}


Result with 16SPP is:

As you can see the results are not that bad but there is one main problem: shadows are missing.. tried many combinations but no improvements. There is some error in the algorithm itself. Can you help me understanding why? Thanks in advance.

EDIT: this is my target reference:

EDIT 2: this is what happens in case the depth > 1 check is omitted:

EDIT 3: after fixing the distFromLight and distFromObj I got the following results:

One of the 2 problems as suggested by @trichoplax could be too soft shadows. So I tried to lower the size of the light, increasing his power to compensate. Results are not good and shown here:

Moreover as requested I can post some more code:

bool checkRayLightIntersection(Vec O, Vec D, Vec & hitLightPoint){
//getting the 4 corners of my light
Vec P1 = Vec(lightPos.x - 20, lightPos.y, lightPos.z + 10);
Vec P2 = Vec(lightPos.x + 20, lightPos.y, lightPos.z + 10);
Vec P3 = Vec(lightPos.x + 20, lightPos.y, lightPos.z - 10);
Vec P4 = Vec(lightPos.x - 20, lightPos.y, lightPos.z - 10);

//the majority of the methods first find out where the ray intersects the plane that the rectangle lies on Ax + By + Cz + D = 0
//in our case the equation of that plane is easy -> D = 20

float t = -(-O.y + lightPos.y) / (-D.y);
if (t > 0){
Vec hitPoint = O + D * t;
Vec V1 = (P2 - P1).norm();
Vec V2 = (P3 - P2).norm();
Vec V3 = (P4 - P3).norm();
Vec V4 = (P1 - P4).norm();
Vec V5 = (hitPoint - P1).norm();
Vec V6 = (hitPoint - P2).norm();
Vec V7 = (hitPoint - P3).norm();
Vec V8 = (hitPoint - P4).norm();

if (V1.dot(V5) > 0.0 && V2.dot(V6) > 0.0 && V3.dot(V7) > 0.0 && V4.dot(V8) > 0.0){
hitLightPoint = hitPoint;
return true;
}
}
return false;
}


Instead, the checkIfRayIntersectSomething method is part of the framework itself (it has just another name), it's not done by me but it's tested and working.

EDIT 4: tried to render a small part of the scene with high number of spp. Tried 512 spp for the bottom right corner where should appear shadows according to the target image. Results show that shadows are not there:

EDIT 5:

I hereby post the code used to render the target image, it does at the same time my Sampling() and GeneratePath() methods using a recursive structure.

Vec Sample(vec3 O, vec3 D, int depth)
{
vec3 color(0, 0, 0);
float t;
if (checkIfIntersectSomething(D, t))
{

vec3 I = O + t * D; //get to the intersection point on the object
vec3 diffuse = getMaterial(I);

vec3 L = vec3(-1 + Rand(2.0f), 20, 9 + Rand(2.0f)) - I; //(-1,20,9) is Hard-code of the light position, and I add Rand(2.0f) on X and Z axis
//so that I have an area light instead of a point light
L = normalize(L);
float ndotl = dot(I.getNormal(), L); //the closer the dotProdutc is to 1.0, the more the light and surface face each other
if (ndotl > 0)
{
if (!checkRayLightIntersection(L)) {
float dist = distFromLight(I);
color += diffuse * ndotl * vec3(1000.0f, 1000.0f, 850.0f) * (1.0f / (dist * dist));
}
}

// russian roulette
float Psurvival = CLAMP((diffuse.r + diffuse.g + diffuse.b) * 0.33333f, 0.2f, 0.8f);

// continue random walk
float rand = Rand(1.0f);
if (depth < 10 && rand < Psurvival)
{
//Besides russian roulette, I also do another weight, because rays that go towards the horizon will bring back very little energy
//so I make a random distribution that favours those rays who are close to the normal of the hit point, this is DiffuseReflectionCosineWeighted(). It creates a Random bounce but proportional to N dot R
vec3 R = DiffuseReflectionCosineWeighted(I.getNormal());//there is a weight

float prob = 1.0;
float cosTheta = fabs(dot(I.getNormal(), R));
if (cosTheta > 1e-6) prob = cosTheta / M_PI;

color += diffuse * Sample(I + R * EPSILON, R, depth + 1) * (1.0f / Psurvival); //the cosTheta() of the attenuation of the rendering equation gets simplified with the cosTheta of the "prob"
//the PI of the prob gets simplified with the BRDF where we are using the ideal BRDF = diffuse/PI
}
}
return color;


}

EDIT 6: Trichoplax was right, I had a problem calculating the distanceFromObject and distanceFromLight. After solving this, we realized that some sort of shadows appeared (shown in one of the images above). He then suggested that such a broad light might have been washing out the shadows, and so it was. By reducing the light size and by using a high amount of SPP we realized that shadows are there and visibile: http://imgur.com/y88zgAZ . As a drawback tho, way less rays reach the light, giving back an image with high noise, but this is another kind of problem and I created another question for it. Thanks to @trichoplax and @julien for their help

• I can see shadows. Therefore i do not concurr with your analysis. However secondary light might have too high reflectivity. Could you try using a simpler scene. – joojaa Apr 7 '16 at 13:10
• @joojaa edited the description with a target reference image – Tarta Apr 7 '16 at 13:31
• Using a simpler scene helps you debug the effect. To me it looks like the shadows are there your scene is just not color corrected and you bounce too nuch energy. – joojaa Apr 7 '16 at 13:57
• @joojaa I just posted some of the code because I am using a much more complex framework. That's why I was avoiding rendering more simple scenes, because it requires me some effort finding out how exactly the scene loader works. Ok, may you suggest me code-wisely what I could change to color the scene correctly? – Tarta Apr 7 '16 at 16:27
• Seeing your updated screenshot (3rd edit), is it still rendered with 16spp? With a only small area light, and no bidirectional path tracing or next event estimation, it's expected to have a lot of noise. Have you tried a much higher number, like 1000spp? – Julien Guertault Apr 11 '16 at 1:55

The problem appears to be unintentionally transparent surfaces

Although the image is grainy, it is sufficiently clear to estimate that all of the darker regions are due to surfaces facing away from the light, rather than due to shadows cast onto surfaces facing towards the light. So it does seem that there is a problem, and the lack of shadows is not just due to having soft shadows from a large area light that don't show up well, or excessively reflective surfaces that provide too much ambient light. Somehow light is getting through your surfaces.

The probable source of this problem

Since you check for intersection with the light and the objects separately, the crucial point where this could go wrong is where you check

if (distFromLight < distFromObj)


You correctly terminate the path and return the light colour if this check is true (that is, if the light is closer than the nearest object in that direction). So the problem is likely to be with one of the two variables you are comparing. If they were both correct, then you would see shadows.

These are calculated on the preceding lines

float distFromObj = length(vert.p);
float distFromLight = length(hitLightPoint);


As pointed out in Julien Guertault's answer, these calculations may be redundant.

length(vert.p) should still be available as t, used 11 lines previously in the calculation of vert.p. This not only avoids unnecessary calculation, but eliminates a potential source of error. Then this length is likely to be correct unless there is a problem with checkIfRayIntersectSomething().

This leaves only length(hitLightPoint) as a source of error, unless there is a more subtle problem. It would be useful to be able to eliminate the possibility of problems with length() or hitLightPoint, which is calculated by checkRayLightIntersection().

We need to see the code to be able to help

length(), checkIfRayIntersectSomething() and checkRayLightIntersection() are not currently included with the code in your question, so we can't analyse these. If you are able to share these that would help narrow down the source of the problem.

A simple way to check whether this is the real problem

You could also verify whether this analysis of the problem is correct by rendering a scene which will eliminate the other possibilities (shadows too soft to distinguish, or shadows washed out by too much reflected light from the rest of the scene). If the light getting through the surfaces is the source of your problem, then it will apply even if there is no path from the light to a surface, provided that surface is facing the light. So render a simple scene that has a light with a plane beneath it, and another plane beneath that, separated by a gap big enough to easily be seen. The lower plane is completely occluded by the upper plane, so should receive no light (no soft shadows and no reflected light from elsewhere). If the lower plane is lit, then you know that this is indeed the problem and you can focus your efforts here.

If the lower plane is not lit, then either this is not the problem, or there is a problem with the scale and the occluding plane may need to be at a different distance from the light in order for the problem to show up. You can use your existing scene as a source of estimated distances for which this problem occurs, and you can use inclined planes in order to see the effect for a variety of ratios of object distance to light distance within a single image.

Sharing the images resulting from such simple scenes may help us identify the problem more precisely.

I notice that in a comment on the question you mention that you do not have a straightforward way of generating scenes yourself. Perhaps there are much simpler scenes already publicly available in the same file format, that you can use to test this?

If not, it sounds like the only thing you have control over is the light. In that case, you could try putting the light underground. If surfaces are not transparent, the scene should be uniformly black. If surfaces are being lit despite an occluding surface, you may see evidence for it this way that would be hard to discern when there is lots of secondary light bouncing around.

• your answer was very useful, thanks first of all ! You where right, distFromLight and distFromObj where wrongly calculated. I fixed it. Something has been improved but the problem is not fixed yet. I posted one picture showing the new results. I thought maybe the error relies on too soft shadows due to a too big light, therefore I tried to lower the dimension of the light. No matter how increase the power of the light, but in this case results are bad (posted a second picture about it). Hope it helps you. In the meantime I am working on the .obj loader for a simpler scene. – Tarta Apr 10 '16 at 17:15
• A smaller light means more of the rays will never reach the light, so you will need to increase the number of rays to give a better quality image. This will also increase the time it takes to render. It might be worth instead making the light as large as possible (an infinite or very large plane). This way, if the problem is transparent surfaces, you should see even more evidence for it, with the added benefit that the largest possible light will reduce the graininess of the image. – trichoplax Apr 10 '16 at 22:45
• I've added an additional final paragraph about putting the light underground as a test (see edited answer) that may help if you are unable to change the scene, but able to move the light. – trichoplax Apr 10 '16 at 22:49
• Ok, I followed your suggestion. I tried to put the light beneath the scene and nothing is lit up. No light passes through surfaces and therefore surfaces are not transparent. At least we can exclude this possibility.. so it can be that shadows are too soft or washed out by too much light reflected from the rest of the scene. How do you suggest me to proceed? – Tarta Apr 11 '16 at 13:06
• Have you tried rendering just a small section of the image yet? For example, the bottom right sixteenth of the image contains are strong shadow in the target image. Rendering just that small image would allow you to use a much higher number of samples per pixel, to give a clearer image and more idea of what is going on. – trichoplax Apr 11 '16 at 14:03

is it correct having such a big light?

I don't see any problem with having a big area light. That said, it also depends on the scale of your scene. If the light is large compared to it, shadows will tend to be more diffuse, like under an overcast sky.

shadows are missing.. [...] Can you help me understanding why?

I haven't found any blatant mistake, but there are two elements that look suspicious to me.

if (depth > 1 && checkRayLightIntersection(O, D, hitLightPoint)) {
// ...
return true; //light hit, path completed
}


Although this part seems to be correct as far as I can tell, I would try to make sure there is no bug introducing a bias. For every bounce, it checks if there is a ray hitting the light first, and if so it breaks out of the loop: in case the code is incorrect and it happens to hit the light more often than it should, the shadows will disappear.

Moreover, in the first if:

return false; //nothing hit, path non valid


and the related loop:

for (int y = 0; y < SCRHEIGHT; y++) for (int x = 0; x < SCRWIDTH; x++)
{
Path path;
if(generatePath(x,y, path)){
Sample(path);
}
}


Make sure you still take the paths that hit nothing into account when averaging the color of the pixel, otherwise this will also introduce a bias.

As joojaa said in the comment, you should try using a simpler scene: a few spheres or cubes, no texture. You can also reduce the number of bounces to one or two to see only direct lighting. If necessary, maybe even use a constant sky dome first instead of a light area.

Other remarks:

struct Path {
PathVert verts[MAX_DEPTH]; //maxDepth is 15 for now
int vertCount;
int x, y; //which pixel this path is referring to
};


Do you need to store x and y?

float distFromObj = length(vert.p);
float distFromLight = length(hitLightPoint);


Don't you already have two t that you can compare?

• First of all thanks for your answer :) So, the method to check the intersection between the ray and the square light is right. I have already been asking about it cause of a small error I couldn't find. The method and the relative solution can both be found here: stackoverflow.com/questions/36180741/… I also edited the question adding a picture that shows the result in case I omit the depth > 1 check. Maybe it can help understanding where the problem is.. thanks once more – Tarta Apr 7 '16 at 19:52