6
$\begingroup$

TWO PART QUESTION - We've all seen the movies/TV shows where the police/feds/spies use computer software to take a grainy photo and do a "clean-up" to see a better picture and more details. I assume the concept is executed by some sort of uber-power pixel-smoothing or anti-aliasing type of algorithms to fill in the blanks based on deductive processing.

PART 1: How real is this technology in the public/commercial software world? I'm not asking about any speculation on alleged secret gov software or such, I just want to know where we actually are with this concept today? How much is fully automated vs human-assisted.

PART 2: Assuming there actually is some reality with this technology for photographs the second part of this question is how (if at all) has this been applied to videos? Again the issue of fully automated vs human assisted is of interest here.

At the heart of this post is the ultimate question of how viable is today's software for being able to take an old VHS or DVD recording and process the frames to create a new HD-resolution remaster. Considering that doing this would mean cleaning up tens-of-thousands of frames for even a simple wedding video I am not expecting this technology to be fast of course.



NOTE: Per in-topic and meta discussions I went ahead and cross-posted this in two other suggested SE forums to acquire their viewpoints and expertise on this matter. So far (it is still a little "early in the day" so to speak) I have received some pretty interesting information in 2 of the 3.

When this is all done (when good answers have been selected) I would appreciate a way to merge these for the benefit of all three SE communities:

$\endgroup$
  • 1
    $\begingroup$ This question is about image / signal processing, which is not within the scope of this site. However, there is a SE for signal processing here: dsp.stackexchange.com $\endgroup$ – RichieSams Jan 8 '16 at 20:41
  • 2
    $\begingroup$ Although this was with good intentions, note that cross posting on several SE sites is not recommended. $\endgroup$ – trichoplax Jan 9 '16 at 16:00
  • 1
    $\begingroup$ Also cross posting is considered rude. $\endgroup$ – joojaa Jan 9 '16 at 16:16
  • 2
    $\begingroup$ You were not asked to post you were told there are other avenues by one user. Cross posting is not the stackexhange way never. If you have a question you need to make it worthy of the site your asking. So you cant ask the same question on video for example. $\endgroup$ – joojaa Jan 9 '16 at 16:22
  • 1
    $\begingroup$ @O.M.Y. the meta post I linked to about cross posting shows community consensus is against cross posting, but there are also answers in favour of cross posting in certain rare circumstances, which are worth reading. Even then they recommend tailoring your question to different sites rather than just copy and pasting. $\endgroup$ – trichoplax Jan 9 '16 at 16:23
2
$\begingroup$

The closest thing I know of to the "computer: enhance!" trope in real life is the family of Single Image Super-Resolution techniques. That page shows a number of examples of the results on various images. You can see that while it improves the visual quality of the enlarged images, it's a long way from what you see on TV where they can read the text on a letter that's reflected in a wine glass, or recognize the face of someone standing in the shadows, etc.

The technique basically works by observing that a single image often contains recurring instances of the same patterns or structures. When you have multiple copies of the same pattern, they often occur at different sub-pixel offsets, i.e. aligned differently to the pixel grid, which means that each copy contains slightly different information about the underlying pattern, and by putting them together you can recover (really, guess at) a higher-resolution version of that pattern. This can then be used to "fill in the missing detail" wherever that pattern occurs. This is a very handwavy explanation, but you can see the paper (linked from the above site) for more details. Most of it is automated, but I'd guess it still requires a fair amount of human parameter-tuning to get good results.

The paper is from 2009, and it looks like there have been a few follow-ups since then, but only incremental improvements.

Also note that if you have a specific use case such as reading vehicle license plates, I'd guess it's possible to use machine-learning techniques to get much better results than you can in general. In that case, you have a concrete, known set of possible shapes (the characters that can appear on a license plate, in the particular font used for license plates), and rather than "enhancing" an image of literally anything, you're just trying to find the set of characters that best matches the image you're looking at. That's not my area of expertise, though.

$\endgroup$
  • $\begingroup$ I haven't heard of anyone doing it yet, but I really think temporal techniques could be used to get sub pixel accuracy from video streams. $\endgroup$ – Alan Wolfe Jan 9 '16 at 2:18
  • $\begingroup$ @AlanWolfe Definitely. Googling "video super-resolution" turns up a number of papers on that idea. $\endgroup$ – Nathan Reed Jan 9 '16 at 4:53
  • $\begingroup$ I did toyed with SISR a while ago, but my mind cleverly decided to erase that information from the brain. If I can recall though one important bit (mentioned in the paper) that I would explicit in your excellent answer is that the patterns or substructures are to be searched not only in the original image but also in multiple scales of it. That IIRC lead to significantly better results, although as I said I don't remember that much details :( $\endgroup$ – cifz Jan 9 '16 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.