I would like to know if it would be possible to use an arduino to run some kind of program to get super resolution of two camera sensors located in close proximity to each other, with the goal being to get a higher resolution output image after both camera outputs are fed into some kind of processing algorithm. Here is more information:
I have a question regarding if anybody has any ideas on how to combine two digital camera feeds together to achieve a near real time super resolution, from two camera modules placed right next to each other, or slightly distanced. I have done some research on this topic , perhaps not enough, however I could not find a solution that I would be able to replicate without an extensive background of technical knowledge.
I am trying to combine two digital camera feeds from two flir lepton 3.5 thermal imaging modules to achieve a higher resolution for a hobby first person view use case, or possibly other digital output cameras. I have found a version of the flir lepton that outputs an analog signal, would this perhaps be an easier start to super impose the images with some kind of image processing to achieve a higher resolution. Also, latency would seem to be a concern, as well as the board used for micro uav platforms under the 250gram range. I know my question if a little out of order in terms of what Iām trying to achieve, I would just like to know if anyone has any pointers or could possibly provide assistance in this matter. This idea/ possible project would be for an rc fpv use case, so I wanted to know if there are any lightweight preexisting products that could achieve this (although I presume they would have to be custom made or have a high expense), with low latency of a couple milliseconds), or if a raspberry pi, or similar board, could achieve this.
The basic concept of increasing resolution with two cameras is flawed. If that is what you want, too bad because it isn't going to happen. Even if you did have technical knowledge.
With world class AI and imaging software, you might be able to achieve a very slight sharpening of the image. But it wouldn't be worth the enormous effort and cost vs. a single camera with greater resolution.
Consider, for example, the camera positions. Each camera actually sees a different image because it is located in a different place. That is just one problem.
The concept of interferometry works for astronomy because the vast distances between the antenna array and the observed object mean that the multiple observations are from virtually the same perspective, whereas what you want to achieve has the cameras a great distance apart, relatively speaking, even if the lenses are touching, because of the short focal length.
You can use multiple-exposure tech to lower noise in the image, you can use it to create a wider dynamic lux range, but you cannot use it to make a higher resolution image if the lenses are more than a fraction of a degree apart.
You can take 4 and stitch the images together. This is not super-resolution or any other such, but it will allow you to get a higher resolution. Stitching will take time and an asic or fpga to achieve real-time results under 10ns, for images of any decent resolution.
I suggest to experiment with merging two images from ordinary cameras using a PC (e.g. a webcam on a laptop) and make sure you understand whether your idea works and merits investing more time, before starting to think about MCUs.
The algorithms used in the closely related topic of photo-stitching are well documented, with research papers available on line.
Understood. I am trying to get a higher resolution video from two modules, would there be any near real time stitching for digital outputs into one output. Ignoring the refresh rate of the lepton at 9hz, would it be possible
Hmmm. A lot of repetition going on here. Do the images overlap, or cover different areas of the image target? The first case is a merge or layer, the second is a stitch. Not the same.
This is the part that is going to block you. Of course it's all possible. Almost anything is possible now. Perhaps you've heard of "deep fakes"? But that doesn't mean it's within any person's reach to develop it themselves.
It depends on what you mean by higher resolution.
The module you speak of has a resolution of 160x120 @ 8.6fps.
You could stitch those together to make either 320X120 or 160x240.
If you used the 24-bit color resolution, you would need to read 115,200 bytes from the two modules. After reading them you would need to merge the buffers line by line, into a third larger buffer, then send that buffer to a display.
It's quite possible. A 32-bit processor should do the job. A multicore would be better. One with an SPI bus for each core would be supreme.
Well, there is no doubt that you can improve the image that way. Look into how some astronomical photos are taken, amateurs are combining dozens of images to make one. It's just, it will only make it very marginally sharper.
What about mechanical dithering? You could vibrate the camera or move it a few microns between a series of shots with the same camera. Oh, I forgot... it's live video. Well maybe the vibration and a bit of weird processing, IDK.
I don't even think that's needed.
You could probably get away with an ESP32 if the buffers for the images are like bitmaps.
How long does an SPI read take for 19,200 24-bit words? That times two.
Read each line from those and pair them into a wide buffer or read each image buffer entirely into a stacked buffer. With two cores addressing the same RAM like the ESP32 has, it shouldn't take too long.
Then push the bigger buffer to a display, 38,400 24-bit words.
Sounds like the start of a plan!
What is your approach for aligning and overlapping the images?
How would you solve and correct for the wide angle lens distortion?
What type of image interpolation do you favor?
I'm sure the OP could use some guidance on these rather major issues.
In terms of compensating for the standard wide angle lens, I was thinking of using a telephoto lens for the two sensors to get a higher crop for alignment. Thank you to everyone who responded so far.
Have you thought about just doing interpolation with one camera only, so it looks like you have better resolution...like this Adafruit example for the AMG8833?
raw 8x8...very crude:
interpolated...very cool:
from:
That should be so much simpler, and considering that the Lepton Flir accuracy is +/- 5-10 Ā°C, well, even if you could combine two images in real time, what would that get you?
