6
$\begingroup$

How do I generate bokeh that simulates shallow depth of field given a perfect depth map?

Here's what I tried:

  1. Generated a sample scene using blender and corresponding depth map

enter image description here

  1. Using Python, generated what sigma (standard deviation) to use at each pixel for gaussian blurring: This one makes it look like the far end of the pink cube is perfectly focused.

    sigma_map = np.array(np.round(abs(np.array(depth, dtype='float32') - preferred_focus_depth)).clip(0,255), dtype='uint8')
    

enter image description here

black = 0, no blur; white = 255, large blur

  1. Applied gaussian blurring (using the sigma_map) at every pixel and this is what I got

enter image description here

Flaws observed: Averaging kernels just near the edges are taking in the pink color nearby hence the pink bleed. How do I avoid this?

Note: It's not completely blurred look closely at the zoomed in image on the right. The pixels that I wanted to remain untouched are still untouched (even though it appears they were blurred as well because of the blurring near the edges)

This blog on Portrait mode on Pixel 2 says, "Actually applying the blur is conceptually the simplest part", is there any simpler way of doing it then?

$\endgroup$

migrated from stackoverflow.com Jun 22 '18 at 9:58

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Since there is no code or anything it's kind of hard to give advice. My best guess is that in your gaussian blur for the pixels above/behind the pink cube you still sample from all neighbouring pixels - including the pink cube. Thus, it blurs with a pink color. So in your kernel you need to disregard neighbouring pixels, if they come from a non-blurred object. $\endgroup$ – Tare Jun 22 '18 at 10:17
  • $\begingroup$ Sorry for not posting any code: didn't add it here because I was just doing plain old convolution but with different kernels for each pixel. "So in your kernel you need to disregard neighbouring pixels, if they come from a non-blurred object": How do I do this? $\endgroup$ – Saravanabalagi Ramachandran Jun 22 '18 at 12:29
  • $\begingroup$ That depends on what you're working with. If this is a glsl shader, then you probably have what you called sigma as a texture available and read from a rendered image to write into the output. So rather than reading from neighbouring pixels of your sigma texture and adding them to your output, you could multiply the the pixels color with the sigma value before adding it to the output. Since sigma=0 for the perfect focus, this would stop the pink color from leaking to the background. Note that you need to adjust the kernel normalization though, otherwise the picture darkens around the cube $\endgroup$ – Tare Jun 22 '18 at 12:34
  • 1
    $\begingroup$ Have a look at this blog post: Bokeh depth of field in a single pass by Dennis Gustafsson. It goes through the development of a bokeh-from-depth filter step by step, including various artifacts encountered with simpler implementations, and how to fix them. $\endgroup$ – Nathan Reed Jun 23 '18 at 21:30
5
$\begingroup$

The blog post that you talked about, is not about generating bokeh for a computer generated image. It is instead about generating a believable depth of field effect from an image captured by a smartphone camera, as the effect is desired for portraits to make the subject stand out. It generally works by splitting the image in to two parts. One part is the subject, which should not be blurred. The other part is the background, which should be blurred but the pixels of the subject should have no influence and would thus have a weight of zero. Slightly more sophisticated algorithms may make the blur kernel size variable across an image to create a smoother transition, but the subject's pixels would still be ignored.

In games and CGI, post process depth of field is done differently. There are multiple ways for it. A simple google search shows multiple. You could do depth of field like you have, and simply ignore the pixels which should not be blurred to remove the ghosting. However, this is not true to how depth of field occurs inside of a camera. So, in order to get a good algorithm to simulate depth of field, I will first try to explain how you get the depth of field effect in an actual camera.

Let us say, that we have an object, a single lens and an image sensor. We look at them from the side. enter image description here Light scatters in many different directions when hitting the object. To simplify we only take one point. From this point, I will show the light that would hit the lens as a gray solid. enter image description here The lens bends the light towards a single point. That is the focal point. enter image description here You can see that the focal point is in front of the sensor. This causes the light from that single point on the object, to fall on a large portion of the sensor. If the focal point is behind the sensor, the same would happen. But, if the focal point lies on the sensor, there would only be one point on the sensor where the light falls on. This area where the light falls on the sensor, is called the circle of confusion (CoC). This is also the cause of the depth of field effect.

One important thing to know, is that when the point on the object moves around, the focal point also moves around. If the point moves closer to the lens, the focal point moves closer to the lens, which would alter the size of the circle of confusion. This gives you the gradual change in blurriness that you have with depth of field, since the circle of confusion gradually becomes larger when an object moves away from the area in focus.

If we add in an aperture we simply block out a part of the light, which causes the circle of confusion to be smaller. This allows us to determine how much light to let in (to make the image brighter or darker), and how strong the depth of field effect is. enter image description here

From this, we can conclude a few things on how we should implement depth of field.

  • Every point whose light enters the camera lens, has a circle of confusion.
  • The circle of confusion is the area where the light, from that point, falls on.
  • The circle of confusion changes size when the point moves closer to or further away from the lens, or when the aperture changes size.
  • The area of the circle of confusion has no falloff. Every point inside the circle of confusion has the same weight. This results in a box blur.
  • If a point is in focus, its circle of confusion would be either zero or something so small, that it is not noticed.

The algorithm is quite simple. For each pixel in the image, we use the depth to calculate the circle of confusion. Then, we add the colour of that pixel to all the other pixels that lie in the circle of confusion. However, if that other pixel is closer to the camera (a lower depth value), it would block the light from our pixel, and we should not add the colour of our pixel to the blocking pixel.

In pseudo code it would look something like this;

// for each pixel in the all-focus image
for(x = 0; x < width; x++){
    for(y = 0; y < height; y++){
        // How much blur? Zero or close to it if it should be in focus.
        float CoCRadius = calcCoCRadius(depth(x, y));

        // For every pixel in the square with the size of CoCRadius.
        for(i = x - CoCRadius; i <= x + CoCRadius; i++){
            for(j = y - CoCRadius; j <= y + CoCRadius; j++){
                // Discard pixels outside of the shape. Square -> Circle
                if(isInBokehShape(i - x, j - y)){
                    // Discard pixels that would block it.
                    if(depth(i, j) >= depth(x, y){
                        // Add the colour.
                        newColour[i, j] += colour(x, y);
                        // Add the weight, for normalizing.
                        weight[i, j] += 1.0f;
                    }
                }
            }
        }

    }
}
//Normalize the values with the weight.
for(x = 0; x < width; x++){
    for(y = 0; y < height; y++){
        newColour[x, y] /= weight[x, y];
    }
}

This would give you a depth of field effect that is quite true, and does not give you the ghosting effect. $calcCoCRadius$ is a method that calculates the radius in pixels of the circle of confusion. It could be anything, but if you want it as true to an actual camera, then you can use the formula from this Wikipedia page. $isInBokehShape$ is a method which returns a boolean on whether the point is inside the bokeh shape. With the two for loops we just get a square, but if we want to get a circle shaped bokeh, we need to discard the pixels that would lie outside of the circle. This discarding is also the reason why we keep track of the weights and normalize afterwards with the weight, since we do not know how many pixels we need to discard. The weight is also useful if you do want a falloff in the circle of confusion, for example if you want a gaussian blur.

Another important thing to know, is that the implementation listed here, is not very optimal. The problem is basically that you write to multiple different pixels. This causes race conditions. Instead, it is better to rework the algorithm so that you only need one write. Basically, you loop through every pixel and you then you check all other pixels (or pixels inside of a maximum CoC radius) whether their CoC radius is large enough that they contribute to the pixel's colour. This way, you can also put the normalizing of the values inside the first loop over all the pixels instead of having to do that later.

This should be a fairy good way of implementing depth of field as a post processing effect. There are also other ways, but this is one.

$\endgroup$
  • $\begingroup$ According to the above formula, pixels with lower depth value will not get colors from nearby pixels at higher depth value (i.e, no bleeding of bg into objects). However, after some careful thinking and experimenting, I found that colors don't bleed only near the focus plane, and hence for a plane even a little away from focus plane, color of the object bleeds into the background and vice versa. Pictorial explanation here. How do I account for that? $\endgroup$ – Saravanabalagi Ramachandran Jul 3 '18 at 11:53
  • $\begingroup$ @SaravanabalagiRamachandran The algorithm I gave already has some of this effect, namely the part where the foreground bleeds on to the background. However, the part where the background bleeds in to the foreground (because the foreground only becomes part of the light that would hit that particular pixel) is not in the algorithm. This would make it a lot more complicated for only a small detail. Something that should also be considered, is that you do not know enough information (you only know the closest object's color) to correctly do DoF, so you have to make some compromises. $\endgroup$ – bram0101 Jul 3 '18 at 17:07
  • $\begingroup$ Doesn't the CoC radius represent the region in which a pixel distributes its energy instead of the region from which that pixel gathers energy? So shouldn't the isInBokehShape use the CoC radius of pixel (i,j) instead of (x,y)? $\endgroup$ – Matthias Aug 26 '18 at 16:57
  • 1
    $\begingroup$ @Matthias Yes, that is actually correct. I based the algorithm on the Camera Lens Blur in After Effects (but added the depth check), which does it using $(x,y)$ instead of $(i,j)$. Although, I do think that in most cases it would not really matter, unless you are going for the most physically accurate in which case you would be better off ray marching the DoF against the depth image. $\endgroup$ – bram0101 Aug 26 '18 at 17:20
-1
$\begingroup$

Does the effect need to be physically accurate? I've implemented this using a cheat before. @bram0101's answer is great. If you need accuracy, then by all means, implement that. I've been able to get something convincing to the eye by simply doing a threshold of the luminance of the image and placing a semi-transparent polygon or circle centered at every point that's above the threshold.

You also need to take into account distance from the focal plane. The distance can be used to control the size of the bokeh, with the size being 0 at the focal plane, and some maximum size at some farther distance.

$\endgroup$
  • $\begingroup$ I didn't get that completely, is it luminance (intensity?) alone or depth-aware as well? Do you have any pseudo code or some code available. $\endgroup$ – Saravanabalagi Ramachandran Jun 23 '18 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.