Reading JPEG with Arduino & Project Ideas

I originally started with the idea of a GLaDOS type robot (if you've played Portal)

This design was pretty static, so I figured I could have two ethernet webcams (stereoscopic vision) plug into a router which could then go to my PC. However, I have since given more thought to making the robot mobile, meaning the robot could move out of range of my laptop. With the computer doing the processing, I originally thought about some point cloud stuff, but since I haven't come across any good resources for how to do that (I would like to work in Java if possible), I thought I'd go more for a mapping of the environment, so making rectangles in space, x distance away from the robot based on the distance sensor, painting the square with the image from the webcam, and then displaying the 3D environment in an interactive app on the PC (so the user could fly around the area the robot has mapped out). However, if the robot is moving around, I then have a desire to have the robot be capable of doing some, at least minimal, image processing on its own.

So I'm envisioning that the ethernet shield be commanded to access the ethernet camera jpg url, store the jpg on the SD card (I have half a gig of storage), and then the Arduino could filter the jpg data off the SD card and look for key colors, or try to do some primitive comparison between the images to attempt to figure out where it is in the room, or how far objects are away, or to figure out if anything is moving, etc. Also once the robot got within range of my laptop, I could download the image data (with a location and distance for each image) and add it to the 3D environment kept on the computer. I'd like to be able to read the raw rgb data from a jpg image using the Arduino, do you have any recommendations for resources to use? I doubt I'd have the stamina to code a codec for jpg image processing from scratch in C, so recommendations for premade software modules or hardware that does the processing are appreciated

Also any other ideas extensions to the project are welcome. I've given some thought to getting a GPS unit - if the robot is roaming the nearby area, I'd prefer to give it a better sense of where it is (and allow it to find its way back home). I haven't looked into the internal time keeping of the Arduino, but I can sync a time piece when the robot connects with the laptop. I'm also interested in the robot being able to climb stairs, the quicker the better, though I haven't really worked out a design yet, I'm hoping for something that weighs approximately the same as a medium sized cat when the batteries are included (essentially looking to make a robotic pet).

My parts list: 2 ethernet cameras http://wpcdn.padgadget.com/wp-content/uploads/2011/01/dlink-cam.jpg 1 8-port (and wireless) router http://www.stardi.com/wp-content/uploads/2010/08/D-Link_DIR-632_Wireless_N_8-Port_Router.jpg Arduino Mega 2560 Ethernet Shield PING distance sensor HM55B Compass IMU/temp https://www.aeroquadstore.com/ProductDetails.asp?ProductCode=SEN-10121 servo http://www.hooked-on-rc-airplanes.com/images/servo.jpg 1 std motor http://img.diytrade.com/cdimg/472172/3499190/0/1224408480/Small_DC_Motor_for_electrical_tool_Toy_Hobby_Massage_machine.jpg 1 Amp H-bridge

I am proficient in Java, this will be my first real project working with C/C++

I'd like to be able to read the raw rgb data from a jpg image using the Arduino, do you have any recommendations for resources to use?

There isn't any RGB data in a JPEG.

RGB is an acronym for the the red, green and blue values that are stored in an image file, each trio of numbers represents the color of a pixel on the screen when the image is displayed. JPEGs are compressed to save space, so this data cannot be accessed directly, thus my inquiry into this project.

If you do not feel that JPEGs retain RGB data and instead use CMYK or some other such raw storage type after decompression, please enlighten me. Simply stating that an image file contains no color data makes me think you're trolling.

JPEGs are compressed to save space, so this data cannot be accessed directly, thus my inquiry into this project.

Probably many ways to do what you want, converting the jpg into a bmp file first then searching the file for your info might be one way. large file handling operations on an arduino might take a long time to complete. You might also need to evaluate if the SD card and ethernet shield operations can be performed at the same time (they may use a common pin preventing concurrent operation).

ParallelLogic: I'd like to be able to read the raw rgb data from a jpg image using the Arduino, do you have any recommendations for resources to use? I doubt I'd have the stamina to code a codec for jpg image processing from scratch in C, so recommendations for premade software modules or hardware that does the processing are appreciated

I don't know about the stamina, but I definitely know about the knowledge: At the moment you have your head in the clouds, tossing around buzzwords an imagining that only because the buzzwords fit into the same sentence the whole thing exists in reality. Lets poke some holes into those clouds and bring some reality in:

The Arduino mega256 has 8kb of RAM, that's for heap and stack plus a little for interrupt tables. Decoding JPG in the amount of memory is barely possible if at all, doing image processing isn't.

The Arduino works at 16 MHz. Realtime data from web-cams: Forget it. There was a reason why in the late 1980ies all the home-computers couldn't do even simple video or audio processing. The Arduino is about as powerful as those computers. Think Commodore64 or Apple II with no RAM and you get close.

So if you want to go ahead with your project, forget the Arduino and get some embedded Intel-board or and ARM processor. The Arduino is just a micro-controller for simpler tasks.

Korman

First apologies for being so short; I was on my way out and didn't want you to waste any time digging further.

Enlightenment:

Not only does a JPEG not contain any RGB data, it doesn't contain anything you might recognise as a pixel. (pretty contentious, huh?)

You were quite correct that the colour space is not RGB, but it isn't complementary colour (CMYK) either, it is YUV. However, the compression in JPEG does not take place in the spatial domain of even the YUV space (all the YUV space does is put most of the image information into the Y component, which is what the human eye is most sensitive to), it takes place in the frequency domain. JPEG compression is pretty much symmetrical (unlike MPEG) in that the compression is as computationally intensive as the decompression.

So, starting with the source RGB image, the image is transformed into the YUV space by a per pixel matrix multiplication. Once in YUV, 8x8 blocks (8 x 8 x 2 bytes) of each of the components is subjected to a Discrete Cosine Transform. This is like half an FFT, producing an 8x8 block of the spatial frequencies in the image, but chucking away the phase information. Then the frequencies in the 8x8 block are quantised, and a zig-zag pattern is trailed through the block, noting run-lengths of similar values. The run lengths are Huffman coded. DC components (the top left element of the 8x8 block) are treated slightly differently and are delta coded. The resulting bit stream is written to the file, along with special marker codes to mark out where the tables are stored for the Huffman codes and the quantisation values. There are usually different Huffman codes and quantisation values for the Y component and the U and the V.

It is possible to do all this in an Arduino, but reassembling the image (assuming you have enough RAM to hold the tables) is all but impossible in a reasonable timescale. The Arduino could never hold anything but the lowest resolution image.

Like a Turing Machine, everything is possible, but you'd grow a beard waiting for just a thumbnail to decompress, unless you knew a lot about the file format in advance.

Lets poke some holes into those clouds and bring some reality in

I think I'm falling in love with Korman.

At the moment you have your head in the clouds

I'll grant you this was a very ambitious thought, I was merely hoping to build off someone else's work should there be any. Even working with a small portion or even a few pixels would be useful, but building off of FFTs and and any inverse/correlation to decode images combined with matrix math does appear to be far over the processing capabilities of the Arduino. I was not aware how extensive/complicated the JPEG decompression process would be, thus my inquiry. Given I've seen BMP work done with the Arduino, I thought there may be some hope for a JPEG application

tossing around buzzwords an imagining that only because the buzzwords fit into the same sentence the whole thing exists in reality

Any time you display a JPEG image on your screen, the JPEG is being broken down into component pixels, I don't think it's a question of if it exists in reality, only whether or not the Arduino can replicate that function. I don't know what terms you feel I am treating as buzzwords, I would think RGB would be pretty straightforward.

You might also need to evaluate if the SD card and ethernet shield operations can be performed at the same time

I was doing some brief research on that, still am, I got as far as the SD card work is done via one pin and ethernet communication is done via another pin. That would likely mean I'd need to route the info through the Arduino rather than saving it directly to the SD card, which could again not be feasible (I suspect the ethernet shield only allows for reading images over the connection and not writing data). I did note some others were looking into this as well. If nothing else, I'd hope that I could at least save the position/distance/orientation sensor data to the SD card, though including the images I feel would be a great asset to my project.

forget the Arduino and get some embedded Intel-board or and ARM processor

That may be what I end up doing way down the line

pretty contentious, huh?

Somewhat surprising, though given the level of compression, I figured some high level encoding was occuring

but chucking away the phase information

I find this intriguing, I normally think of the phases as being just as important as the frequencies involved in terms of being able to reconstruct a signal. I appreciate the depth of the response you've provided, this definitely gets me off on a great start to getting a better grasp of what's going on with JPEGs.