Confusion about HDR image pipeline

Question

I am working on a path tracing renderer, and I want to produce an sRGB image out of the HDR image buffer I get at the end of my rendering algorithm. Up until now I have worked just by clamping values, which of course works fine only for a limited range of light conditions.

Now, reading around (I started here, then read some of the links given in that post) I got that one way to go is (starting in linear RGB space):

1. Calculate the exposure
2. Scale all the values in the image by the exposure, channel by channel
3. Apply some tonemapping function
4. Gamma correct.

The preferred way to calculate the exposure seems to be as the inverse of "the luminance value that will saturate the sensor", computed from some average luminance value of the image; the latter in turn is obtained by computing a histogram of the log2's of all pixel's luma, ignoring some percentage of values (e.g. 50% of the darkest pixels and 5% of the brightest), and computing the average of the remaining values.

Another way I found was in this answer here, where the scalar factor seems to be based only on the pixel value, rather than the whole scene. EDIT: Here, it seems to me that the steps 2 and 3 above are replaced by a different procedure:

• The color gets mixed to its luminance by a factor of $\left(\frac{m-0.18}{m}\right)^{20}$, where $m$ is the maximum value in the RGB channels
• Tonemapping is applied only to $m'$ = ($m$ after the above mix)
• The color channels are multiplied by a factor equal to $f(m')/m'$, where $f$ is the tonemapping function.

1. What's the theory behind the second approach, and does it relate in any way to exposure?

2. Is it normal that if I apply the first method, I usually (regardless of the percentage thresholds I use) get images that are way darker than if I apply the second (although maybe a bit more realistic), or is this a smell that I'm getting something wrong in my implementation of the first method? (Additionally, the images I get by rendering my test scenes with Cycles are as bright as the images I get from my renderer using the second method, thus not nearly as dark as the ones I get from the first method.)

Here are some examples: for reference, this is a test scene I'm using, rendered with Cycles, containing two emissive squares each of emission respectively 2, 10, 100 and 1000

Here is the same scene, with the same factors, rendered with my renderer, applying this tonemapping function directly to each color channel, after exposure-correcting (histogram-based, clamped at 50% for dark and 5% for light)

Here are the tests without exposure correction (same tonemapping):

Here, I am using the answer's method to deal with the alternative "scaling" (with the same tonemapping function I used for the results above).

The exposure-corrected gives clearly wrong results, as the 10, 100 and 1000 values all give pretty much the same output, the non-exposure-corrected one has some weird saturation values but it's perceptively the closest to Cycles' illumination levels; the answer's method flattens the background too much, and feels a bit darker than it should, at least compared to Cycles' output.

At this point I'd guess that I'm applying the exposure correction wrongly. A simplified version of my code is the following:

constexpr float a = 2.51;
constexpr float b = 0.03;
constexpr float c = 2.43;
constexpr float d = 0.59;
constexpr float e = 0.14;

float expsr = this.exposure();

// the image buffer contains RGB values (no alpha) in linear space
for (size_t j = 0; j < this.image_buffer.size(); ++j)
{
    image_buffer[j] *= expsr;
    image_buffer[j] = (image_buffer[j] * (a * image_buffer[j] + b)) / (image_buffer[j] * (c * image_buffer[j] + d) + e);
}

And afterwards I gamma-correct. Am I supposed to use the exposure value or the tone mapping function differently?

Answer 1

Note that the other answer that you linked to is describing only a tonemapping operator, and does not include any exposure scaling at all. It's not replacing your steps 2 and 3 in the process, but only step 3. So it's potentially confusing and not all that useful, I think, to compare images that are using an exposure estimation + tonemapping pipeline with images that are using only the tonemapping part.

I think of exposure estimation and scaling as being a separate part of the image pipeline prior to tonemapping (and not part of tonemapping). The purpose of exposure is to bring the image values into the range where most of the detail we want to see is roughly in the [0, 1] range. The exposure-scaled image is still HDR and may have values above 1, but we aren't going to be able to see as much detail in the values above 1 after tonemapping.

One reason we need to do exposure mapping could be about the units of measurement, but another reason is just that natural scenes have wildly varying general light levels. A scene in daylight might have its overall intensity values 100x greater than a scene at night. If you try to put both through the same tonemapping curve without any exposure scaling, either the daylight scene will come out completely washed out and you won't be able to see anything, or the night scene will be practically black and you won't be able to see anything. By using exposure scaling we can adjust these scenes to a reasonable range before tonemapping. To put it another way, with exposure we are selecting what range of values contains the detail we want to be visible in the final image.

Methods of estimating exposure, like log-luminance histograms and such, are more associated with real-time rendering scenarios like video games, where the user might move their camera around the scene and sometimes be looking into dark areas, sometimes into bright areas, and the game needs to adapt the exposure to make sure they can see something. There's a lot of tuning and tweaking needed to get such algorithms to behave reasonably and not be distracting.

However, it is also perfectly acceptable to just tune the exposure value by hand when you have a shot you're composing. You can just set it up so that it looks "right" to your eye. You can also make artistic choices, like for example slightly overexposing a scene to communicate brightness/heat/harshness, or underexposing to make something feel more mysterious/foreboding, etc.

The exposure-corrected gives clearly wrong results, as the 10, 100 and 1000 values all give pretty much the same output

I think you figured this out already, but this is exactly the expected result from exposure scaling if you're setting exposure based off the luminance of the scene. Scaling all the light sources in the scene 100x brighter results in the exact same image only with all the pixel values 100x brighter, which results in an exposure scale 100x smaller, which results in the same final image.

OK, now let's talk about tonemapping. Usually tonemapping curves are described as a scalar curve: one input, one output. Broadly speaking, two general approaches are to either apply this curve to each of the R, G, B channels individually (evaluating the function 3 times per pixel), or to apply the curve to the overall luminance of each pixel only (evaluating it just once per pixel).

The per-channel approach has the nice property that it automatically desaturates colors as they get brighter, which is aesthetically appealing: a bright, colored light source may appear as a white core with colored bloom/glare around it, which does a good job of making it "feel" really bright to our visual system. In the per-channel approach, this happens because of the shoulder of the tonemapping curve, where it approaches its horizontal asymptote: the difference between R, G, B values gets compressed, producing an output that's closer to monochrome.

However, the per-channel approach also tends to mess with hues, as after going through this nonlinear curve there is no guarantee that the final RGB color represents the same hue as the initial one. In order to fix this, you can calculate the luminance of the input color, apply the tonemapping curve to that, and then scale the input color to match the new tonemapped luminance. However, this approach doesn't alter the saturation of the input at all, so you lose that desirable property of the per-channel approach.

The other answer you linked is working around that by putting the desaturation effect back in by hand. The author came up with this $\left(\frac{m-0.18}{m}\right)^{20}$ function that you mentioned as an empirical estimate of how much desaturation there should be. The mix from the original color to its luminance just implements the desaturation.

A somewhat more principled approach to this could be to base the desaturation off the slope of the tonemapping curve, as that's where the effect originates in the per-channel approach. But tonemapping curves are completely empirical and based on aesthetics anyway; there is no real theory behind them that I'm aware of. So ultimately you can do anything that works and produces images you like.

Answer 2

I think I got at least some of it. I'll write here what I've found out thus far, but I'll keep the question open for a few weeks, in case someone comes up with more insight.

I'll refer to the book "High dynamic range imaging" by Reinhard et al. as [HDRI].

I'll start with my confusion about exposure (what I'm about to say only applies to ideal cameras, the matter changes significantly if one wants to model a realistic camera). According to [HDRI], $\S$ 7.6.1 the "exposure" correction is only needed when the units of the scene are not specified/agreed upon. In this case, the scalar correction before tonemapping is aimed at "guessing" a plausible reference for the value of the mid grey, and correcting accordingly. Therefore, for a ray tracer, it's sort of a design choice whether to assume that the units of the lights in the scene are to be interpreted as standard radiometric units or not. If not, the exposure correction is needed, but then the behavior shown in my question's tests is correct/what one should expect. This accounts for the exposure-corrected example (2nd set of tests).

As for Cycles and my non-exposure-corrected one, in both cases (at least, I'm guessing this is the case for Cycles, I didn't carefully check the code) the tonemapping curve is applied right away to the color channels (or better yet to multiply all the channels by the ratio between the tonemapped luminance and the original one, see [HDRI], $7.6.2$; which is at least part of what happens in the last method). The differences should mostly be due to the different curves used in the two renderers (I guess Cycles' one is not filmic, at least by default(?)).

Finally, regarding the last method/set of tests, it's still not entirely clear to me. I get the $f(m)/m$ factor is just a more numerically stable way to apply the tonemapping function to the color channels, but the rest of the method is still kinda mysterious. I guess the idea is to scale color values, similarly to what one does with the "exposure method", while at the same time applying a saturation trick to bake in the "high values means bright scene" assumption. I guess is sort of a compromise solution between assuming the units and just guessing the scene brightness globally from some average. I'd still be interested in knowing where the formulas for this trick come from though, I didn't manage to find anything yet (I still don't have the reputation to comment under that answer and ask for references).