What is the state-of-the-art on using computers to "clean-up" images?

Question

TWO PART QUESTION - We've all seen the movies/TV shows where the police/feds/spies use computer software to take a grainy photo and do a "clean-up" to see a better picture and more details. I assume the concept is executed by some sort of uber-power pixel-smoothing or anti-aliasing type of algorithms to fill in the blanks based on deductive processing.

PART 1: How real is this technology in the public/commercial software world? I'm not asking about any speculation on alleged secret gov software or such, I just want to know where we actually are with this concept today? How much is fully automated vs human-assisted.

PART 2: Assuming there actually is some reality with this technology for photographs the second part of this question is how (if at all) has this been applied to videos? Again the issue of fully automated vs human assisted is of interest here.

At the heart of this post is the ultimate question of how viable is today's software for being able to take an old VHS or DVD recording and process the frames to create a new HD-resolution remaster. Considering that doing this would mean cleaning up tens-of-thousands of frames for even a simple wedding video I am not expecting this technology to be fast of course.

---

---

NOTE: Per in-topic and meta discussions I went ahead and cross-posted this in two other suggested SE forums to acquire their viewpoints and expertise on this matter. So far (it is still a little "early in the day" so to speak) I have received some pretty interesting information in 2 of the 3.

When this is all done (when good answers have been selected) I would appreciate a way to merge these for the benefit of all three SE communities:

• Computer Graphics: What is the state-of-the-art on using computers to "clean-up" images?
• Signal Processing: https://dsp.stackexchange.com/questions/28168
• Video Production: https://video.stackexchange.com/questions/17363

Answer 1

The closest thing I know of to the "computer: enhance!" trope in real life is the family of Single Image Super-Resolution techniques. That page shows a number of examples of the results on various images. You can see that while it improves the visual quality of the enlarged images, it's a long way from what you see on TV where they can read the text on a letter that's reflected in a wine glass, or recognize the face of someone standing in the shadows, etc.

The technique basically works by observing that a single image often contains recurring instances of the same patterns or structures. When you have multiple copies of the same pattern, they often occur at different sub-pixel offsets, i.e. aligned differently to the pixel grid, which means that each copy contains slightly different information about the underlying pattern, and by putting them together you can recover (really, guess at) a higher-resolution version of that pattern. This can then be used to "fill in the missing detail" wherever that pattern occurs. This is a very handwavy explanation, but you can see the paper (linked from the above site) for more details. Most of it is automated, but I'd guess it still requires a fair amount of human parameter-tuning to get good results.

The paper is from 2009, and it looks like there have been a few follow-ups since then, but only incremental improvements.

Also note that if you have a specific use case such as reading vehicle license plates, I'd guess it's possible to use machine-learning techniques to get much better results than you can in general. In that case, you have a concrete, known set of possible shapes (the characters that can appear on a license plate, in the particular font used for license plates), and rather than "enhancing" an image of literally anything, you're just trying to find the set of characters that best matches the image you're looking at. That's not my area of expertise, though.