What is bilateral upsampling?

I've worked with various interpolation algorithms including linear, cubic (and bilinear and bicubic), other bezier (and nurbs, etc) based interpolations, lagrange interpolation and some others, but I haven't been able to find any basic info that explains what bilateral upsampling is.

I have been able to find information on Joint Bilateral Upsampling which as best as I can tell interpolates multiple pieces of data in parallel, using hints from each to help the other interpolate better, but am having trouble separating the info of BU and JBU.

Is anyone able to explain the basic ideas of what bilateral upsampling is about?

It seems like you're asking two things. I can't really speak technically about JBU, but I can give an overview of the necessary concepts and bilateral filtering generally. You'll probably need to find more details yourself, but this should give a coherent structure to start from.

Fixing "Image"s

Many image-processing people view filtering as either something to be done as a post-process, or as a way to scale images. The mathematician's perspective is more accurate, descriptive, and complex.

When you have an image file, you don't have a real image. You have pixel "values", which are strictly valid only at the center of pixels. Roughly speaking, each pixel has an area that takes on that color. Mathematically, what you're doing is using a nearest-neighbor reconstruction filter to reconstruct the true image from pixel values. So an "image" is really a set of samples and a reconstruction filter, typically nearest-neighbor.

Resampling

When you upscale an image, what you're really doing is resampling. How do you do this? You take your signal, and generate a new set of samples from it. But for an image, I just said we don't have the real image; we have to reconstruct it first. So a typical resampling procedure looks like this:

2. Reconstruct an image between the values, which live at pixel centers, by using reconstruction filtering.
3. Measure a new set of samples from the reconstructed image, possibly spaced differently.
4. Throw away the old samples and use the new ones.

The key point is that the reconstruction filter used is the "filter" used for resampling. For example, a "bicubic upsample" would use a bicubic reconstruction filter.

It turns out that if you use a sinc function for the reconstruction filter, and you don't shrink the image, you get (theoretically) the same image back. In fact, if you upscale the image and then downscale it again, you'll get the same answer. This isn't generally true for any other filter.

Most filters approximate a sinc filter to more or lesser degrees. Deviations result in image loss.

Filtering

If you use the same before/after sampling rate, you do something called "filtering". Formally, filtering is resampling without a change in sample rate. Filtering is a special case of resampling.

The only possible purpose to filtering it to throw away information (e.g. a Gaussian) or possibly also to distort or add spurious information (e.g. a Sobel).

Bilateral Filtering

You can use many different filters for filtering. Bilateral filtering is one of them. I don't believe it's considered state-of-the-art for any purpose now (though I don't know what is; probably some newfangled machine-learning thing I expect).

A bilateral filter is a non-linear filter, meaning it doesn't have a nice Fourier representation. Conceptually, it is like a Gaussian, but the source colors themselves also factor into a pixel's "distance" from others. This tends to smooth flat areas while preserving edges.

Bilateral Upscaling

By now, I hope you see where I'm going with this: any filter you can use for filtering, you can use for upscaling or downscaling, which are really the same thing (because all of it is resampling).

If you've implemented a bilateral filter, you should be able to rather simply implement a bilateral filter resampler - just as, if you've implemented a Gaussian filter, you should be able to implement a Gaussian filter resampler.

The only difficulty may be the definitions. Unfortunately, the distinction between filtering and resampling is systemically muddied to occasional non-existence in the image-processing literature. For implementing a bilateral-type filter, I would rely heavily on its similarities to Gaussian resampling. IIRC (I haven't read the paper in years) bilateral filtering is posed in terms of pixels, not samples.

• How bilateral filtering specifically works is what I am trying to get information about. Can you elaborate on that? Jun 20, 2016 at 21:01
• One can lend images for the explanations in the beginning from here Jun 20, 2016 at 21:10
• @AlanWolfe I assumed you were asking after broader context, since implementing basic bilateral filtering is highly Google-able and rather simple. As I wrote, the basic idea is to make values factor into the weight, not just distance. Jun 21, 2016 at 3:49
• Also @joolaa's answer is quite nice and may be useful as a second explanation of my overview. Jun 21, 2016 at 3:49
• Now that I have the term "bilateral filter" I am having more luck on Google. Thanks for that. Jun 21, 2016 at 3:53

This probably does not explore the full depth of the term, but the first thing that comes to my mind when I hear "bilateral upsampling" is depth-aware blending of low-resolution images onto high-resolution ones; for instance, when you render alpha-blended geometry to a half-resolution buffer (for performance savings) and then composite it back onto the main render target. Angelo Pesce wrote a post that explains it well.