0
$\begingroup$

I'd like to know if the following technology already exists, and if not, how feasible it would be to create something like this:

I came up with an idea for creating a higher-res image that is put together from multiple lower-res images. More specifically, from a video (animated, multiple frames) source. The best way to explain it would be to explain what would happen using an example:

Let's say that there's video footage from a security camera of a van driving by. The van has a logo on it. Based on the entire video (using all applicable frames to gather data), object recognition code is used to recognize that an object has moved in the video. Using the raster data in the frames along with the object recognition, a vector is generated that represents the moving object. Interpolation can be used to show where this object is during all timeslices, including those between frames. So for example, if played in slow motion, you would see (just for example's sake) a vector rectangle moving along with the van's logo (let's say the logo is rectangular just to make the example simpler to conceptualize). As the frames play by one at a time, the rectangle smoothly follows the logo, always matching the location of the logo in each frame, but smoothly moving between those locations between the rendering of each frame, yet always matching the location of the logo when each frame actually shows up (let's say we play it at 1fps, to better imagine what's happening). This interpolation would allow for the tracking of the object based on limited data (frame rates can't be infinite, so picking any timestamp between the frames would allow interpolation to determine where that logo would be between the frames, based on the motion that the frames depict).

Having explained the situation above, now let's apply the general idea of the interpolation but in a different way..

The rectangular vector is split up into a grid. This grid can represent a raster image. However, its resolution would be higher than that of the video's resolution - the higher-res more-detailed version of the logo that we're trying to capture. So now, let's build the high-res logo...

Finding the first frame of the video in which the logo is visible, each pixel of the higher-res logo is filled in based on the pixels of the logo in the video's frame. When processing the first frame, the resulting higher-res logo looks the same as the logo in the frame. The only difference is that it looks like a blown-up pixelized version of the logo in the frame. So far, no 'enhancement' is done.

However, as we go further frame-by-frame, we now have some extra data to work with. Even though the video has limited resolution, with each pixel representing a blend of all the colors within its bounds, the fact that its motion is being tracked allows each subsequent frame's representation of the logo to slightly add some more detail to the higher-res logo. Since the location of the logo as it's moving across the video is interpolated, the higher-res pixel grid can be placed between pixels (using floating-point coordinates), as it's able to determine a more exact placement of where the logo is, beyond the resolution of the video.

With each subsequent frame, the visible pixels of the logo gradually add more and more detail to the higher-res logo, eventually making it look less and less pixelized. With enough video footage of the moving logo, it could be possible to eventually generate a much more detailed version of the logo than any one of the frames can individually provide.

I hope that my explanation of this tech was sufficient to provide an idea of what I'm trying to describe. You may have to extrapolate and interpolate this imaginary concept in your mind and intuitively fill in some details (similar to what this tech attempts to do), to fully conceptualize how this tech would work.

Having explained it, I'd like to know if something like this exists. If it does, what is this kind of technology called? And if it doesn't, how difficult would it be to create something like it? I imagine that utilizing machine learning would make it even more effective in filling in details.

If you've read this far, thank you for your time. I'd like to know what you think.

$\endgroup$
5
  • $\begingroup$ The idea is nice, in principle it should work... my worries are at the detection part. If your image is a low res picture, then you are limited by finding the positioning of the logo... Therefore you wouldn't find the correct pixel position of your high res logo. In case your position and orientation capturing is perfectly accurate, you could add a list of values per pixel, average them and later make a convolution with a sharpening filter. Be in mind, I am absolutely a beginner! So maybe this comment is absolutely useless ;) $\endgroup$ – Thomas Mar 24 at 10:34
  • $\begingroup$ This is already a thing, most notably it is done with high end telescopes. The big idea here is signal averaging. Just align the images, add them together and take the average. The noise tends to average out of the image resulting in a clean crisp image. Image alignment is also a thing, like software that stitches images together for panoramic shots (that's just one example). I don't have any specific examples for you but I know there are libraries just for this sort of thing. $\endgroup$ – pmw1234 Mar 24 at 11:09
  • 1
    $\begingroup$ We did this in a tracking application where we tracked the deformation of metal. And by applying this kind of technique could measure our seed positions at a higher res than the resolution of the camera $\endgroup$ – joojaa Mar 26 at 15:26
  • $\begingroup$ take a look at this: projectphysx.epizy.com/superresolution.html $\endgroup$ – Thomas Mar 28 at 13:48
  • $\begingroup$ This is also a thing in film restoration. Vendors who make restoration software are understandably quiet about the specifics of their algorithms (a lot of it is understanding the nature of specific film artefacts such as grain), but prior to the deep learning era, a lot of it was believed to be based on optical flow and expectation-maximisation algorithms. $\endgroup$ – Pseudonym Apr 13 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.