file processing/transfer

Projects General Guidance
1

Basically what I want to do is use a arduino Leonardo (atmega32u4) as a pass through processer for a camera/photo data.

I'm well aware that it can't handle real time video processing and buffering for display. The idea is to use compressed data protocols (video over USB serial) or use more time to get the job done.

  1. However considering webcams tend to use USB serial to transmit the video (compressed), in theory the chip should be more then capable of taking that serial signal and then writing it to say a SD card, or to a XBee module for a device that can.

  2. Or just want to take a picture and save it to a sd card, I don't care if it takes 2-5/10+ seconds for it to process the pixel data.

  3. Severely reduce the pixel count/quality so it can do real time video for say motion detection.

Many of these applications would probably use processing(.org) on my computer, all I really need is a means to transfer the data in the correct formats.

In regards to #1 I doubt its so simple that I can just read the bytes coming in. If I just wanted to watch it from a third device as a stream that might work, but if I write the bytes to a SD I would need to make it into a .mov or similar for watching on a computer at a later time. Sounds like its should be possible but how?

Any coding applications in arduino for these kinds of cases.

2

Sounds like its should be possible

Not to me it doesn't.

3

harddrive123:
in theory the chip should be more then capable of taking that serial signal and then writing it to say a SD card, or to a XBee module for a device that can.

I don't know what theory leads you to that conclusion but I think you're wrong. Even though it's possible to stream video over USB and it's possible to connect USB to an Arduino, it does not follow that an Arduino can accept or generate a video stream.

You haven't said what you're trying to achieve, but if you are looking for a wireless video transmitter I suggest you buy one.

If you want to manipulate the video stream in any way, do it on a PC or some other platform that has similar capabilities.

4

Not to me it doesn't.

That's nice to know, but telling us you don't know either isn't what I was asking.

I don't know what theory leads you to that conclusion but I think you're wrong. Even though it's possible to stream video over USB and it's possible to connect USB to an Arduino, it does not follow that an Arduino can accept or generate a video stream.

I think I was pretty clear, I'm trying to use arduino as serial/i2c/... pass through device or buffer if you will. Its compressing data, I'm not uncompressing or trying to display anything. The camera sends video over serial, arduino reads the bytes from its serial0 and then sends the exact same bytes somewhere else over serial1 (Leonardo) to a computer or to a SD card.

A serial port is a serial port regardless of what its attached to assuming they match in speed(Baud rate) and protocols used which they do. If one can send there is no reason why the other can't accept unless there unpaired which means you never had a serial port open to begin with.

The concept is that you do something like
x = Serial.read(); // read a byte of data from camera
Serial1.print(x); // Write that byte to computer/xbee/ or to SD card

Is there code that can automatically mirror the incoming data/bytes to a different port/serial to reduce the time lag and system resources?

If I want to make a .mov file out of my video on a sd card it needs a file header and end (file start at memory address 0xXX, write frame data, end file), it can be done, but is there code for it in arduino?
That way if I take the SD card out and put it on my computer I see a file, not blocks of unformatted frame data.

The memory buffer isn't large enough for a whole frame of video (at least for most resolutions), but this is compressed data so it shouldn't be an issue.

If that truly is impossible (with reason not nay say) how about just taking pictures.

Clearly its an advanced application. If I had a thousand free hours I could research the protocols in depth and learn more about port coding and timing's but alas I'm just asking if anyone has something already done on the topic.

5

The functionality you describe of taking bytes from an input serial port and writing them to an output serial port is certainly feasible. Your code fragments show roughly how to do it. If the serial port settings are identical then the microcontroller is actually not achieving anything except introducing delay and potential errors into the serial bitstream; you might as well eliminate the Arduino and connect a piece of wire from the input to the output. If you want that async serial bitstream to be carried over USB then you need a USB TTL adapter. You could use an Arduino's integral USB adapter for that but you might just as well use a separate adapter - they're readily available and only cost a few quid.

The problem comes when you say that the data is coming from a video camera and assert that "the chip should be more then capable" of handling the input data stream because it can handle async serial data and async serial data can be carried over USB and the video stream can be carried over USB. It's akin to saying that you can plug an electric heater into a wall socket, and you can charge your phone from a wall socket, therefor it must be possible to power an electric heater from your phone. Not everything you plug into a wall socket is compatible with everything else you can plug in. Not everything you can plug into USB is compatible with everything else that can be plugged in. In particular many USB devices use much higher data rates than the Arduino could ever handle.

Take the video/image data stream out of the equation and and make it somebody else's problem to present your data as an async serial stream and the Arduino can read and write that. It can also perform any transformations you want on that data stream, subject to the extremely limited memory and processing power available. I suspect you'll find that getting a video stream onto an async serial stream is rather challenging, though.

If you want more specific objections other than "the Arduino can't do the sort of thing you're trying to do" then you need to be more specific about what your inputs and outputs are going to be; if you think you have the basis of a solution and want feedback of whether it's realistic, tell us what solution you have in mind. At the moment, I don't see any.

6

Be aware that the fastest an Arduino (except the Due) can communicate is 8Mbps (that's mega-bits per second). That works out to 1 megabyte per second, for one direction. Copying out the other side cuts the available bandwidth in half. Overhead will also reduce throughput. I'm guessing we're in the realm of 128 kilobytes per second.

That's SPI. I2C and Serial are much slower.

Throwing a $35 Raspberry Pi and a $20 USB webcam at this problem makes much more sense. The hardware is orders of magnitude faster, and the drivers and image processing software have already been written. You just need to glue the pieces together with a Bash/Python/Ruby/etc script.

7

harddrive123:

Not to me it doesn't.

That's nice to know, but telling us you don't know either isn't what I was asking.

Sorry I was not saying I don't know, I was saying that you haven't got a hope in hell of doing it. Your thinking is very arm waving without a hint of putting any figures into your thought.

Just for one moment consider the data rate you are asking of the devices and work even the most elemental thing out.
What is the data rate from the webcam? What baud rate does this translate into. Can the webcam buffer the data because sure as hell the arduino can't. Can you write to an SD card fast enough and sustained enough with an arduino?

8

Lets ask a simpler question.

Does anyone know the code require to initialize a webcam to have it start sending data? Processing has code that can do it cam.start() once you set make the cam the capture. I'm looking for code that can do the same with an arduino.

I know for a fact that people have been able to do very low resolution video play back with arduino. How does one do that?

The camera has feedback, it generates the frame sizes based on the resolution your using.

Although the serial buffer size is a difficult problem to work around I highly doubt its a dead end. Just don't let the buffer get filled. Wait for the start of the frame then keep reading/writing until its done. I saw a video where a guy hooked up a camera to a arduino and had the arduino connected to the computer, using what looked like a simple 3rd party image software he was able to load a picture from the camera through arduino to his screen. It took 2-3 sec for it to load but it clearly worked, only problem is it gave no code and poorly written write-up.

Alternately you can just ignore part of the data from the camera, only read 1 in 3-10 frames sent. Or in the best case, you make the camera a slave device, and it will only send frames at request.

I could spend a few 100 and just buy the hardware that can do it with ease but that's not the point.

I'm basically fishing for ways to use the camera with the atmega chips, simply saying it can't be done is a half lie. It can't be done to that specification (video processing), so what does one have to cut away to find a specification that will work?

9

harddrive123:
Does anyone know the code require to initialize a webcam to have it start sending data? ... I'm looking for code that can do the same with an arduino.

Depends on the camera. There are many different modules out there, each with a different protocol.

http://www.arducam.com/ might be a good place to start.

10

simply saying it can't be done is a half lie.

No it is more like one hundredth of a lie.

I see there is little point in offering yo advise as you seem determined to know better.
I wonder why you asked it in the first place.

I have done extensive work with video in the 80s with processors of the same power as the arduino, you will be lucky to get a frame every 4 seconds.

11

harddrive123:
Lets ask a simpler question.

Does anyone know the code require to initialize a webcam to have it start sending data? Processing has code that can do it cam.start() once you set make the cam the capture. I'm looking for code that can do the same with an arduino.

It's a simpler question but actually a lot harder to answer. Do some Googling on USB protocols to get a feel for the complexity involved in using USB. USB very cleverly hides the end user from all sorts of complexity when plugging USB-shaped devices into each other, but to achieve that the USB protocol itself ends up being extremely complex with all sorts of negotiation and discovery going on. Every time you plug a new device into a PC the PC has to go find the right software driver to provide the PC's end of that USB protocol and present the data from that device in a form that makes sense to the PC. In the case of a video camera that means creating a virtual imaging device which fits into the imaging framework on the PC and integrates with codecs and stuff to manage the optical qualities and encoding and all sorts of stuff. And again, at the other side of the OS your application can just open a virtual video source without knowing how it works, and rely on standard video I/O libraries to control the video source and manage the streams it provides. The role that Processing plays in all this is minimal.

If you want your Arduino to take the place of the PC then you need to discover and understand how the video camera expects to interact with the PC (the USB protocol it uses) and implement the host part of that protocol. Then you'll need to recreate the PC functionality to manage that camera and the video streams it's providing. Except of course that the Arduino hasn't got a hope in hell of handling the USB video stream from a video camera because the video will consume massively more USB bandwidth than the Arduino could ever dream of handling.

If you want to receive a video stream from a USB camera, connect it to a PC. There are PCs around which are comparable in size to an Arduino and provide a real OS with all the drivers and runtime frameworks to deal with this job, and the resources to deal with the resulting data. The Arduino basically doesn't have a chance, it is completely the wrong platform for the job.

12

Grumpy_Mike:
No it is more like one hundredth of a lie.

I see there is little point in offering yo advise as you seem determined to know better.
I wonder why you asked it in the first place.

I have done extensive work with video in the 80s with processors of the same power as the arduino, you will be lucky to get a frame every 4 seconds.

If I thought you actually had advice on the topic I might feel disappointed.

Just for one moment consider the data rate you are asking of the devices and work even the most elemental thing out.
What is the data rate from the webcam? What baud rate does this translate into. Can the webcam buffer the data because sure as hell the arduino can't. Can you write to an SD card fast enough and sustained enough with an arduino?

The webcam is serial, it has a serial chip, the webcam has a default baud rate because that best matches its data rate to video compression. Thus if you have a serial connection that means the webcams baud rate and the computers/or other serial devices baud rate are equal and they agree to the same packet packaging protocol. If the webcam wants to use a faster baud rate then what the other device can handle, the webcam will use the slower baud rate to match the other device or the connection will simply be rejected. This is what slows down the fps the most, the camera wanted 30fps, but the data rate can only handle ex.2fps, the extra frames that don't sync up with sending just don't get sent.

Now I haven't confirmed this but I'm 99.999% certain that read/write operations from RAM memories and even external memories (SD card) have far smaller latencies then a serial connection at any baud rate within the accepted standard of TTL serial. (at least in most cases, I bet hardware from the 80's would be slower thus creating timing mismatch, guess that's our misunderstanding)

If you don't program correctly and allow the serial buffer to be filled, you will get data lose not slower transfers, its just shift registering the data through. Serial is interrupt based, it should be easy to prevent buffer overflow. If not, why?

I don't really expect an answer mr.grumpy, I've found some interesting examples else where and I'm going to go play around with the hardware/coding and see what I can get from it.

13

harddrive123:
The webcam is serial, it has a serial chip, the webcam has a default baud rate because that best matches its data rate to video compression. Thus if you have a serial connection that means the webcams baud rate and the computers/or other serial devices baud rate are equal and they agree to the same packet packaging protocol. If the webcam wants to use a faster baud rate then what the other device can handle, the webcam will use the slower baud rate to match the other device or the connection will simply be rejected. This is what slows down the fps the most, the camera wanted 30fps, but the data rate can only handle ex.2fps, the extra frames that don't sync up with sending just don't get sent.

There is a world of different between an async serial connection and a USB connection. The quote above gives the strong impression that the camera supplies an async serial interface and NOT a USB interface. However, your original post strongly gave the opposite impression:

harddrive123:
considering webcams tend to use USB serial to transmit the video (compressed), in theory ...

Which is it? Do you have a link to the camera spec?

14

harddrive123:
The webcam is serial,

I would be very, very surprised if you can find an RS232 (TTL or otherwise) camera module. RS232 usually isn't pushed beyond 115200bps because it's not terribly reliable at higher rates, and 115200bps generally isn't enough for video nor even still frames of any significant size or resolution. Consider a 640x480x8bpp picture, which has dreadful image quality: in raw format that requires 2.4MB of data. That amount of data will take about 26 seconds to download at 115200bps. Most people are not willing to wait 26 seconds between shots, so there is no incentive for hardware manufacturers to make RS232-interfaced cameras today. (You might be able to find one made in the 90's though.)

If the webcam wants to use a faster baud rate then what the other device can handle, the webcam will use the slower baud rate

One problem with slowing down the transfer is, with most cameras made today you are reading the CCD directly, so if it takes 2 seconds to read a frame, you'd better hold the camera perfectly still to shoot nothing but still lifes because the 2-second shutter-"open" time will make any movement a blur.

Some camera modules have built-in SRAM (I believe the Arducam has this), so the "shutter" is only open a few ms and then you can download the stored freeze-frame at your leisure. You probably want to look for this type of camera.

EDIT: Well I guess I get to eat crow: http://www.evola.fr/product_info.php/en/linksprite-jpeg-color-camera-ttl-interface-p-172 Looks like it only supports still frames, though (no video). The JPEG is nice, as it makes the amount of data that needs to be transferred smaller, but it makes it a lot more difficult to do any kind of motion detection since you have to decode JPEG and you don't have a lot of RAM for storing the decompressed image data.