HacroCam - recalculate to greyscale before sending (processing example)

Hi,

I got a HacroCam. It works fine with the examples provided. I want to use the processing example.
As I in the end want to work with a relative slow wireless serial connection, I would like to recalculate the picture to greyscale before transmitting it.
The formula is simple GRAY = 0.2989 * RED + 0.5870 * GREEN + 0.1140 * BLUE. This should save 2 byte per pixel.

I tried digging through the code of the HacroHub.ino and the Processing Hacrocam Demo.
Unfortunately I didn't get too far.

I someone has a HacroCam and has more experience than I have .. I would be very glad.

Thanks
Robert

first note that in your formula
GRAY = 0.2989 * RED + 0.5870 * GREEN + 0.1140 * BLUE
the weights do not add up to 1.000 (make red 0.2990 to solve)

There are faster formulas (which are less accurate) if you are in a need for speed.

The most easiest formula is
GRAY = GREEN;
as green is in the middle of the spectrum. Did this trick once for a real time color2grey video stream (only use the green signal).

Another one seen sometimes is the simple average:
GRAY = (RED + GREEN + BLUE)/3;

or with integer weights - which may be faster than floating point
GRAY = (3 * RED + 6 * GREEN + BLUE)/10;
GRAY = (5 * RED + 10 * GREEN + 2 * BLUE)/17;
GRAY = (26 * RED + 51 * GREEN + 10 * BLUE)/87;

One can prevent an expensive division by using "ideal" values which are shifts:
GRAY = (RED + 2GREEN + BLUE)/4; // will be a shift 2
GRAY = (5 * RED + 9 * GREEN + 2 * BLUE)/16 ; // becomes a shift 4
GRAY = (19 * RED + 38 * GREEN + 7 * BLUE)/64 ; // becomes a shift 6
GRAY = (77 * RED + 150
GREEN + 29 * BLUE)/256 ; // becomes a shift 8
(above this the multiplication does not fit into 2 bytes anymore)

This last one has an error margin less than 1% per weight.
77/256 = 0.3007
150/256 = 0.5859
29/256 = 0.1133

Do you have a link to the code?

Thanks a lot for your reply. Very good explanation.

The code can be found here:
http://hacromatic.com/page/21

What I use is the HacroHub sketch in conjunction with the processing sketch.
It is quite a lot of code ...

Thanks for your effort.

Robert

The Arduino harcoCAM lib does not contain RGB conversion..

THe processing code has the interesting code at line 370

                r = (r << 3)|(r >>> 2);
                g = (g << 2)|(g >>> 4);
                b = (b << 3)|(b >>> 2);
                img.pixels[loc] = color(r, g, b);

think this is where you can gray-ify the color, something like code below

gray = 0.2989 * r+ 0.5870 * g+ 0.1140 * b;
r = g = b = gray

                r = (r << 3)|(r >>> 2);
                g = (g << 2)|(g >>> 4);
                b = (b << 3)|(b >>> 2);

                gray = 0.2989 * r+ 0.5870 * g+ 0.1140 * b;
                r = gray;
                g = gray;
                b = gray ;

                img.pixels[loc] = color(r, g, b);

give it a try..

And again thanks.

I want to change both sides so that only the greyscale value gets transmitted from the arduino to the PC. The goal is not greyscale in itself but minimizing the amount oft serial communication.
So changing the data stream from the camera on the fly before transmitting it via serial.

I am afraid that it needs some major change in both sides.

Robert

You could first give the above code a try than you know that making the rgb values identical gives you a nice gray.

then patch the download function in HarcoCam.h

    void download (Stream &serial, uint32_t address, byte capturemode)
    {
        setReadAddress(address, TWI_BUFFER_LENGTH);

        byte header[13] = { 0xff, 0xff, 0xff, 0x68, 0xa, 0xc, 0x72, 0x6f, 0x6d, 0xa, 0x74, 0x69, 0xc };
        for (byte i = 0; i < 13; ++i)
            serial.write(header[i]);
        serial.write(hardwareID());
        serial.write(capturemode);

        byte buf[TWI_BUFFER_LENGTH];
        uint32_t numbytes = TotalBytes[capturemode];
        uint32_t blocks = numbytes/TWI_BUFFER_LENGTH;
        byte remainder = numbytes % TWI_BUFFER_LENGTH;
[color=red]        for (uint32_t j = 0; j < blocks; ++j)
        {
            read(buf, TWI_BUFFER_LENGTH);
            serial.write(buf, TWI_BUFFER_LENGTH);
        }
        if (remainder)
        {
            read(buf, remainder);
            serial.write(buf, remainder);
        }
[/color]    }

The red part copies color in RGB values that are in buf to serial.

replace this with something like

        for (uint32_t j = 0; j < blocks; ++j)
        {
            read(buf, TWI_BUFFER_LENGTH);
            changeRGB2GRAY(buf);  // <<<<<<<<<<<
            serial.write(buf, TWI_BUFFER_LENGTH/2 ); ///<<<<<
        }
        if (remainder)
        {
            read(buf, remainder);
            changeRGB2GRAY(buf); // <<<<<<<<<<<
            serial.write(buf, remainder/2); //<<<<<<<<<<<<<< 
        }

changeRGB2GRAY(buf) unpacks 2 bytes in RGB value and calculates 1 byte gray values .

I assume you can write this one, the processing code shows you how to do it.


HarcoCamdemo.pde line 336 allocates total bytes to receive.

__ int totalbytes = imagewimageh2; // remove the factor 2__

Then change this part line 359 ++

        for (;;)
        {
            if (serial.available() > 0)
            {
                bytesread = serial.readBytes(inbuffer);
                System.arraycopy(inbuffer, 0, packedbuf, packedpos, bytesread);
                packedpos += bytesread;
                if (packedpos == totalbytes)
                    break;
            }
            else
            {
                delay(1);
            }
        }
        if (m_debug)
            println("image read complete.");

        for (int row = imageh-1; row >= 0; --row)
        {
            for (int col = 0; col < imagew; ++col)
            {
                int loc = imagew*row+col;
                // This code is verified to be correct.
                int msb = packedbuf[2*loc] & 0xff;
                int lsb = packedbuf[2*loc+1] & 0xff;
                int r = (lsb >>> 3);
                int g = ((lsb & 0x7) << 3);
                g |= (msb >>> 5);
                int b = (msb & 0x1f);

                r = (r << 3)|(r >>> 2);
                g = (g << 2)|(g >>> 4);
                b = (b << 3)|(b >>> 2);
                img.pixels[loc] = color(r, g, b);
            }
        }

to something that reads the gray values

        for (;;)
        {
            if (serial.available() > 0)
            {
                bytesread = serial.readBytes(inbuffer);
                System.arraycopy(inbuffer, 0, packedbuf, packedpos, bytesread);
                packedpos += bytesread;
                if (packedpos == totalbytes)
                    break;
            }
            else
            {
                delay(1);
            }
        }
        if (m_debug)
            println("image read complete.");

        for (int row = imageh-1; row >= 0; --row)
        {
            for (int col = 0; col < imagew; ++col)
            {
                int loc = imagew*row+col;
                img.pixels[loc] = packedbuf(loc);
            }
        }

and it should work ... more or less ...

(disclaimer - no harcoCam experience whatsoever :wink:

no back to the basic question: you want to have it faster.

a quick hack in the HarcoHub.ino

void setup()
{
    Serial.begin(345600);
}

And adjust this speed also in HarcoHub.pde line 533: int g_arduinoSerialSpeed = 345600;

If 345600 does not work you might try 230400; I have worked with both speeds without serious problems.

But it might be that there is a new/other bottleneck now..

Oh man - you are great. I'll try to work out the changeRGB2GRAY function and see if it works.

Changing the serial speed doesn't help me as I will link to the arduino via a 115200 baud wireless link (which in reality will do less).

Thanks a lot for getting me started.
Robert

Sorry to bother you again. The code you provided works great. (one time round instaed of square brackets, but at least this I was able to find XD)
If I don't change values, I get a crippled pic and the time required for the transfer is halved.

So far so good.

Even thought I thought I have enough information to work out the changeRGB2GRAY function, I am stuck right at the beginning.
To keep it simple and lower the risk of failures on my side, I planned to at first not use a function but do it inline. In addition I wanted to submit only "black" instead of doing the recalculation.
Sounds simple...but if I put in the code the function no submittion of the picture takes place anymore. This really puzzels me as it should simply rewrite the first half of the buffer array.
I would love it if I had better debugging possiblities for the controller - or better coding skills.

That's what I did:

//changeRGB2GRAY(buf);  // <<<<<<<<<<<
for (uint32_t l= 0; l < blocks/2; ++l){
buf[l]=0;
}
serial.write(buf, TWI_BUFFER_LENGTH/2 ); ///<<<<<

It would be great if you could help out (and teach me something :slight_smile: ) again.

Robert

OK, at least the simple test works now. I get a nice one colored picture. Even though it is blue instead of grey. But this must be something within Processing. I set createImage to ALPHA. But this doesn't make a difference.

PImage img = createImage(imagew, imageh, ALPHA);

My test code for the arduino library is now:

for (int l= 0; l < TWI_BUFFER_LENGTH; ++l){
buf[l]=125;
}

Not sure why the other code didn't work. But it does now ...

Robert

OK - stuck again. I don't get the bit shifting from Java trnaslated to C++.
I opened a thread in the programming section.
http://arduino.cc/forum/index.php/topic,141274.0.html

You brought me so close .. I thought I would get it. :slight_smile:

Robert

answer in other thread

@Hacrocam users

Attached you find an updated arduino library, the arduhub sketch and an updated processing example for the Hacrocam.
The update allows to convert the image to 8bit grayscale before submitting it via serial link. This halves the time required for the transfer. Great for low baud rate serial links.
In addition you can take more than one picture without restarting the sketch. Press w to take a picture.

The processing example has an added user definable variable g_grayscaleImage to enable/disable grayscale convertion.

Many thanks @robtillaart who solved the hard part. I just put together the puzzle pieces and extended the library and sketches.

Trying to extend it with a 4bit grayscale option now.

Robert

Hacrocam_8bit_option.zip (19.5 KB)

HI,

I did have the same idea yesterday to get a 4bit gray-scale image.

Another idea is to send an image progressively. Suppose you have an image of 8x8 pixels. Then you send the pixels in this order:
// layer 0 lowest resolution (stepsize n/2)
[0,0]
[4,0]
[0,4]
[4,4]

// layer 1 (stepsize n/4 but not those already send)
[2,0], [6,0]
[0,2], [2,2], [4,2], [6,2]
[2,4], [6,4],
[0,6], [2,6], [4,6], [6,6]

// layer 2 (stepsize n/8 but not those already send => which are all remaining pixels for an 8x8 bitmap.
[1,0], [3,0], 5,0], [7,0]
etc

There advantage of progressive images is that you get an idea of the overall picture and it refines gradually.
It also allows to stop transmitting when the resolution is high enough.

Besides the proposed schema, there are many more schemas. The best known is the alternating lines.
First send all the even lines and than all odd lines.

Another speed up of transmission could be run length encoding. This is especially useful if you go to the 4 bit modes.

A byte could become a {run length; colour } tuple, with a run length of 1..15 and a 4 bit colour.
If you have 6 bytes of a certain colour (e.g. colour 9) you send { 6, 9 } or as byte B0110 1001

Hi,

send an image progressively is a good idea. Unfortunately in my case it will not work for two reasons:

The image gets streamed from the camera to the Arduino via TWI/I2C though a 32byte buffer (so 16 pixels).
Only the buffer can be reworked. That's because auf the limited RAM available. The image in total would not fit into the RAM.
I could download the image pixel by pixel from the cam. But this would slow things down again an negate the compression work done.

At the moment in Processing the picture doesn’t get rendered during the download. I tried to change that (to see the pixels arriving) but didn’t succeed. As I am one of the many victims of the “new topic button now showing in the Processing forum” bug, I can’t even post to get help there. ?

The run length encoding is interesting too. But I would lose half of the space for pixels. Maybe it would be possible to mix 4bit grayscale and run length encoding with some signaling byte. But as I can only process 16 pixels at a time, I am not sure if this in reality would improve speed. Taking into account the lost information space and the calculation time.
I looked at other compression methods. But they are all too heavy for an Arduino.

I also looked at other color schemes. 3-3-2 RGB looked nice too. But as far as I read the shading depth is more important for the human brain than the color. (especially for me, as I am partially color blind XD )

If you have more ideas – please through them in. Maybe we get it to stream a video over the 115k baud. ]:smiley:

Thanks
Robert

I tried the 4bit grayscale. The saved trasmission time gets eaten up by the calculation time. It may still be worth the effor, as it will help on speeds slower than 115k baud.

I think I got close but don't get a correct picture. I think the error is on the decoding side. But I don't get it. Maybe if I look at it later again.

(Still l and one char variables in, as it's the old code. The newly added variable names are better. :-))

encoding:

//*pbuf is the pointer to the 32 byte RGB pixel buffer

           void rgbTo4Bit (byte *pbuf, int bufflength){
		for (unsigned int l= 0; l < bufflength/4; ++l){
			int msbLeft = pbuf[4*l] & 0xff;
			int lsbLeft = pbuf[4*l+1] & 0xff;
			int rLeft = (lsbLeft >> 3) & 0x1F;
			int gLeft = ((lsbLeft & 0x07) << 3) | ((msbLeft >> 5) & 0x07) ;
			int bLeft = (msbLeft & 0x1f);
			rLeft = rLeft << 2;
			gLeft = gLeft << 1;
			bLeft = bLeft << 2; 
			int grayLeft=(77 * rLeft + 150* gLeft + 29 * bLeft) >> 8;
			grayLeft = grayLeft >> 4;
			grayLeft = grayLeft << 4;
			
			
			int msbRight = pbuf[4*l+2] & 0xff;
			int lsbRight = pbuf[4*l+3] & 0xff;
			int rRight = (lsbRight >> 3) & 0x1F;
			int gRight = ((lsbRight & 0x07) << 3) | ((msbRight >> 5) & 0x07) ;
			int bRight = (msbRight & 0x1f);
			rRight = rRight << 2;
			gRight = gRight << 1;
			bRight = bRight << 2; 
			int grayRight=(77 * rRight + 150* gRight + 29 * bRight) >> 8;
			grayRight= grayRight >> 4;
			
			pbuf[l]=(grayLeft | grayRight);
		} 
	}

decoding (in Processing):

for (int row = imageh-1; row >= 0; --row)
{
  for (int col = 0; col < imagew; col=col+2)
  {
    int loc = imagew*row+col;
          int LeftPixel= packedbuf[loc] >>> 4;
          LeftPixel= LeftPixel << 4;
          int RightPixel= packedbuf[loc] << 4;
          img.pixels[loc] = color(LeftPixel,LeftPixel, LeftPixel);
          img.pixels[loc+1] = color(RightPixel, RightPixel, RightPixel);
    }
  }

It loos like this: 4bit.jpg Picture
Must be something wrong with the col and row count I think,

Robert

decoding:

for (int row = imageh-1; row >= 0; --row)
{
  for (int col = 0; col < imagew; col += 2)
  {
    int loc = imagew*row + col;
    int LeftPixel = packedbuf[loc] & 0XF0;  // mask the higher 4 bits is enough
    int RightPixel = packedbuf[loc] << 4;

    img.pixels[loc] = color(LeftPixel,LeftPixel, LeftPixel);
    img.pixels[loc+1] = color(RightPixel, RightPixel, RightPixel);
  }
}