Parsing image for processing on Arduino

Hi all, I have the SparkFun LinkSprite camera and JpegTrigger board as a starting point for a project - it simplifies things including getting an image off the camera and onto an SD card, which is great. Unfortunately, the camera encodes the image as a jpeg, rather than a stream of pixel data. Easy for shuttling from the camera to the SD card, but doesn't allow any image processing without first unpacking the encoded image.

I understand that the Arduino is not meant for crazy data-crunching, but what I'd like to do is simply downsample the image to a lower resolution and save to the SD. My first thought was finding a C library like CImg that might work, but they are all way too big. Really this is just creating an ongoing sum and averaging values, then writing back out as a simpler image format like BMP - seems like it should be possible.

Other possibility would be something like the VideoExperimenter shield and an Arduino, since things like blob-tracking have been successfully implemented. I'd obviously need another camera with composite out.

Suggestions? Thoughts?

Suggestions? Thoughts?

Generally speaking, it ain't going to happen. You can google the forum for image processing or sparkfun cam to see previous post on similar projects. I think they were all abandoned.

When you say "unpacking the encoded image" and "simply downsample", what exactly do you mean here?

Are you looking for "JPEG decoder"? This IS quite non-trivial given Arduino's capabilities -see what's needed in JPEG - Wikipedia

Or do you already have some library which gives you access pixel values & you're actually only looking to down-sample. This is fairly simple - see Floyd–Steinberg dithering - Wikipedia

Suggestions? Thoughts?

Forget it, it is not going to happen. There is simply not enough memory to do any Jpeg decoding.
You only have 2K of memory in total so that does not go very far with an image.
Perhaps if you said what you actually want to do rather that ask how to implement what you think the solution should be we could offer more help.

I've got WORKING code to detect movement from a Grove serial camera. That's all it does, nothing more. It can do this at about 10fps on Arduino. It's a little tricky but it's well tested! This method does not work for detecting a laser dot in the dark, due to all the noise and compression artifacts. Instead I did it like this: Change the compression so the jpg file is only 4k. Send it to PC over XBee in 1/2 sec. The Arduino doesn't even see the data, it's sent directly from camera to PC, with Uno sending commands to camera. Using Processing it is only a few lines of code to uncompress the file and access each pixel by x,y. Then I can track the location of the IR laser in the dark at 50m. This is like blob tracking? Except there is only one blob in the image when it's dark. Finished today! Testing and working at 2fps! Is this useful to anyone?

Sure, interested. Care to publish it?

Can detect even the slightest finger movement from 10m. Person walking towards at 50m. Perpendicular path at 100m.
These results require good lighting. It fails when the moving object is the same color as it's background of course.
It works better when there is texture on the background and moving object.
Some parameters require tweaking when it's darker in the room.
A bright window in an otherwise dim room causes problems, the shadows have noise.
It fails to detect a laser dot in the dark, but I've solved that on PC!
It detects a static change to the scene, yet ignores moving shadows by resetting every 200 samples.
It detects fast motion like a baseball being thrown across frame.
Can send the picture to PC for display when detected.

#define FLUSH Serial.flush()
#define WR(s) Serial.print(s)
#define WR13 Serial.write(10);
#define WR32 Serial.write(32);
#define HALT {while(1);}
#define CMD(x,var) for(cii=0;cii<x;cii++)Serial.print(var[cii])
//#define DEBUG1 WR13 for(i=0;i<5;i++){Serial.print( int(Serial.read()),HEX);WR32} WR13
//#define DEBUG2 WR13 for(i=0;i<5;i++){Serial.print(char(Serial.read())    );WR32} WR13
char length[] ={0,0,0,0};
char cmdRST[] ={0x56,0,0x26,0}; //reset then >10ms
char cmdCAP[] ={0x56,0,0x36,1,0};
char cmdLEN[] ={0x56,0,0x34,1,0};
char cmdCON[] ={0x56,0,0x36,1,2};
char cmdGET[] ={0x56,0,0x32,0x0c,0,10,0,0,0,0,0,0,0,0,0,10}; //x=16 10*0.01ms
//x=9:
//0x36,d10-80 56k-8k from CommTool, here 40-255
char cmdCMP[] ={0x56,0,0x31,5,1,1,0x12,4,0x36}; 
char cmd640[] ={0x56,0,0x31,5,4,1,0,0x19,0};
char cmd320[] ={0x56,0,0x31,5,4,1,0,0x19,0x11};
char cmdOFF[] ={0x56,0,0x3e,3,0,1,1}; //[6]=1,0 x=7,30ma,60ma
char cmdBAU[] ={0x56,0,0x24,3,1,0xae,0xc8}; //9600
int i,j,k,last,cii,dlen,cnt=0,a[10];
unsigned int picLen;
boolean same2x=0;

void setup(){
pinMode(13,OUTPUT);
Serial.begin(115200);
CMD(4,cmdRST);delay(500);FLUSH;  //need time to get long string
//CMD(7,cmdBAU);delay(500);Serial.begin(9600);
cmdCMP[8]=200;
CMD(9,cmdCMP);delay(10);
//CMD(9,cmd640);delay(10);  //remembers even with long power off how?  causes image corrupt?
//still in R buffer OK?
}
void loop(){
for(k=0,j=0;j<10;j++){
digitalWrite(13,1);
CMD(5,cmdCAP);delay(60);  //try while instead?
digitalWrite(13,0);
CMD(5,cmdLEN);
FLUSH;  //wrong order on purpose
while(Serial.available()<9) ;
for(i=0;i<5;i++)Serial.read();
for(i=0;i<4;i++)length[i]=Serial.read();
//if(Serial.available())WR("Err");
picLen=((unsigned int)length[2])*256 +int(length[3]);
if(j<9)CMD(5,cmdCON); //delay(10);
k+=picLen/30;
a[j]=picLen/30;
}  //for
WR13
i=abs(k-last)/5-6;
if(i<0)i=0;
WR(dlen=i);WR32
for(j=0,i=0;i<10;i++)
  j+=abs(a[i]-k/10);
j-=50;
if(j<0)j=0;
WR(j/5);WR32

cnt++;
if(((dlen>=5)||(j>=10))&&(cnt>5)) {  //20,cnt>50? 5,2
  WR13
  //getdata(picLen);  
  cnt=0;
  same2x=0;
}
CMD(5,cmdCON);
if(abs(k-last)/5<=2)same2x=1;
if(cnt%200==0)same2x=0; //200
if(!same2x)
  last=k;
//delay(500);
}
void getdata(unsigned int len){
const int BSIZE=64;
int count=len/BSIZE;
int tail=len&(BSIZE-1);
unsigned int addr=0;
cmdGET[6]=cmdGET[7]=cmdGET[8]=cmdGET[9]=0;
cmdGET[13]=BSIZE;
for(int i=0;i<count;i++){
  //cmdGET[15]=10;
  CMD(16,cmdGET);
  delay(10); //10
  if(Serial.available()!=BSIZE+10)HALT
//Serial.read();Serial.read();
//if(Serial.read()!=0x32)Serial.println("Halt?");
  FLUSH;
  addr+=BSIZE;
  cmdGET[8]=addr>>8;
  cmdGET[9]=addr;
  }
cmdGET[13]=tail;
CMD(16,cmdGET);
delay(100);
//Serial.read();Serial.read();
//if(Serial.read()!=0x32)Serial.println("Halt?");
FLUSH;
}

This is processing code, correct? Has this code been tried with a B/W cam?

You can see this is not Processing. That code is much longer. It should work with any Serial JPG camera. There are a few different brands, but the command set in my code is the same. It simply looks for a change in the length of the JPG compression. It does this over 2 time frames, 1 shorter for a fast movement. And longer for a static change. SImple?

You can simplify the program flow by commenting out the last 3 if's in loop().
I added a beep and tested by standing still across the room, then moving.

sbright33:
You can see this is not Processing.

There is somewhat of a disconnect between the above statement and the below statement. Just what does the UNO actually do in your setup?

The Arduino doesn't even see the data, it's sent directly from camera to PC, with Uno sending commands to camera. Using Processing it is only a few lines of code to uncompress the file and access each pixel by x,y.

  1. The posted code relates to the beginning of my initial paragraph, using Arduino to detect motion when there's enough light.

  2. After "Instead" I wrote about using Processing on a PC to detect an IR laser dot in the dark. Sorry for the confusion.

What does the Uno do?

  1. In the 1st case, it does it all, without a PC. Detecting motion, sounding an alarm, printing levels, sending a decent picture only when there is motion. See posted code above.

  2. In the 2nd case, it only sends a small picture. Depending on the PC to analyze it. It can do this at 2fps over XBee remotely 1000' away. The Processing code is long, complex and difficult to read. Will improve it before I post.

sbright33:

  1. In the 1st case, it does it all, without a PC. Detecting motion, sounding an alarm, printing levels, sending a decent picture only when there is motion. See posted code above.

It's a neat way to look for change without actually doing any image processing, but doesn't it mean you have no control over the sensitivity of the detector? Even a single pixel change could trigger it.

Thanks for the compliment! You DO have great control of the sensitivity. Just use a bigger threshold to detect change.

Unfortunately it doesn't work when there are big dark shadows. The change in the noisy pixels is greater than most real changes you'd wish to detect. It also does not work in the dark to detect a laser dot. Turning a laser on and off has no impact on the size compared to the noise. With a 1W IR it looks like a huge spotlight to the camera, but it doesn't help. It would not work great with the wind blowing the trees. There will always be a change, although not as much as you think! If the trees are far enough away, even during a storm, the size does not change. It is ideal for a static, well lit, scene. Then you can make it sensitive enough to detect your fingers moving at 10m. Or less sensitive to detect only when you're walking. You can choose. Listed above are the only limitations I've found.

Next stop is to detect even a few pixels of change during daylight. They must be near each other. In a static scene. Ignoring noise in the shadows. This must be done on the PC. I should be able to detect a powerful IR laser dot during the day with this method. But there's no need for this, as you can detect motion without the laser.

Yikes, sorry all! Haven't been getting emails about replies here - my apologies.

@all:
Essentially what I want to do is just what I posted - downsample an image using the Arduino. I'm building a camera and want it to be lower-resolution than the LinkSprite can do (actually, really low resolution - like 9x9px). I realize this isn't trivial nor a simple process, but that's it. I appreciate realistic feedback, but as has been mentioned in similar posts that have been shot down as impossible, I'd prefer suggestions of possible hacks or innovative solutions rather than flat-out rejection.

@sbright33:
Cool project, but I don't think it will work in this case. If I understand correctly, the code just compares the jpg headers or something like that looking for differences? I'm afraid I either need a lightweight jpg library for microcontrollers (I've had great success with Java libraries in Processing, though I understand the complexity and limitations of the Arduino in this case), or pixel-by-pixel access to the image data.

Perhaps the VideoExperimenter shield is a better choice, since it seems pixel access is direct. Time is not an issue, since I'm fine doing the downsampling after the image is stored on an SD card.

The last possibility is the one I started with - to build my own imaging sensor using old SLR lenses and small LDRs. Not super accurate and the engineering is difficult to get a focused image, etc, but might be another possibility.

Any other suggestions or thoughts?

If you're within 1/2 mile of a PC, you can send the JPG file to the PC, then access the individual pixels. It's only 4K so it takes 0.4 secs to send it over XBee. Processing will let you access the X,Y pixels without writing much code. It decompresses the JPG for you.