large arrays and memory

Projects Programming
1

Hi

TL;DR:

32x16x3 array takes more memory than the same data as a long procedural list of instructions, why?

Long version:

I have a 32x16 LED matrix and I want to display colour images that I create on it, like a colour bitmap.

However, the bitmap draw function with the Adafruit library is monochrome only.

I worked out a way to use python in another software to sample the values of each pixel and literally generate a string of instructions to get the arduino to draw each pixel one at a time. That worked, but unsurprisingly took most of the memory on the arduino UNO and I could only get one image on there.

like this:

matrix.drawPixel(0,0,matrix.Color888(0,0,0));
matrix.drawPixel(0,1,matrix.Color888(0,0,0));
matrix.drawPixel(0,10,matrix.Color888(3,98,82));
matrix.drawPixel(0,11,matrix.Color888(3,98,74));
matrix.drawPixel(0,12,matrix.Color888(2,84,57));
matrix.drawPixel(0,13,matrix.Color888(0,0,0));

Next I tried using a similar method to create a 3D array (rows and columns for x and y, and three values for RGB. Here's a small test one I made manually:

uint8_t myArray[4][8][3] = {
  {{255, 255, 255}, {100, 200, 255}, {255, 100, 200}, {255, 0, 0},{255, 255, 255}, {100, 200, 255}, {255, 100, 200}, {255, 0, 0},},
  {{255, 200, 100}, {100, 200, 255}, {255, 100, 200}, {10, 100, 150},{255, 200, 100}, {100, 200, 255}, {255, 100, 200}, {10, 100, 150}},
  {{255, 200, 100}, {100, 200, 255}, {255, 100, 200}, {10, 100, 150},{255, 200, 100}, {100, 200, 255}, {255, 100, 200}, {10, 100, 150}},
  {{0, 255, 0}, {100, 200, 255}, {255, 100, 200}, {0, 0, 0},{0, 255, 0}, {100, 200, 255}, {255, 100, 200}, {0, 0, 0}},
};

This nearly worked too, except it actually took up more memory than just hard coding every value like before, and when storing a full 32x16x3 array it actually caused the whole thing to fail and the LED matrix went mental.

I am resigned to the fact that I will probably never get more than one image on to the unit this way, but now I am curious as to why the array took more memory and whether there's a way to optimize it. I suspect its to do with how hard coded values get compiled or something? I tried using #define but that made no difference.

I am a total noob to low level programming! It's tricky!!

Thanks

PS here's an image of the first attempt that worked:

2048×1152 347 KB

2

Any reason not to use PROGMEM?

3

32 x 16 x 3 = 1536 bytes - 75% of your memory if you have a uno
Indeed progmem would help (hence The program took less memory space)

4

This is one reason why many such devices include an SD card reader/writer. That way, you can store your images in files on the SD card, and read them out as needed.

5

I am shown for the total noob that I am. Had no idea about the PROGMEM thing.

I see now that the reason it worked the first time was because the hard coded data was stored in the much larger PROGMEM. I didn't even realise the memory was separated like that.

PROGMEM fixed it. Thanks!

I can still only fit three images there though, looks like I need an SD card reader.

6

Ok I thought I had it figured, but no.

The image was all garbled so I tested with a small array. The same array returns expected results when in SRAM but garbled results when in Flash:

const uint8_t myArray[h][w][3] PROGMEM = {

  {{0,0,0} , {100,100,100} , {200,200,200} , {0,100,100} , {100,0,100} , {100,100,0} , {100,100,0} , {10,20,30}} ,
  {{0,0,0} , {100,100,100} , {200,200,200} , {0,100,100} , {100,0,100} , {100,100,0} , {100,100,0} , {10,20,30}} ,
  {{0,0,0} , {100,100,100} , {200,200,200} , {0,100,100} , {100,0,100} , {100,100,0} , {100,000,0} , {10,20,30}} ,
  {{0,0,0} , {100,100,100} , {200,200,200} , {0,100,100} , {100,0,100} , {100,100,0} , {100,100,0} , {10,20,30}}
  };
7

When the array is in PROGMEM you have to use code to access the array. You did not post that code. You do have that code, don't you?

8

nope!

9

Seems this is quite complicated. As I said I have no experience with C++ so I'm muddling my way through. Found some docs on this and trying to make sense of them. If anyone is willing to give me pointers, very grateful.

Here's my code so far, without appropriate methods to retrieve data from PROGMEM. If anyone feels charitable enough to give me pointers, much appreciated! Otherwise, I'll keep chipping away!

const int h = 4;
const int w = 8;
const uint8_t myArray[h][w][3] PROGMEM = {

  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 100, 0} , {10, 20, 30}} ,
  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 100, 0} , {10, 20, 30}} ,
  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 000, 0} , {10, 20, 30}} ,
  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 100, 0} , {10, 20, 30}}
};

void setup() {
  matrix.begin();
  uint8_t a;
  uint8_t b;
  uint8_t c;
  for (a = 0; a < h; a++) {
    for (b = 0; b < w; b++) {
      matrix.drawPixel(b, a, matrix.Color888(myArray[a][b][0], myArray[a][b][1], myArray[a][b][2]));
    }
  }
}
10

There are special instructions (C functions) to read data from PROGMEM. Look at the examples in the reference section.

11

also, may want to consider a board with more memory, like a mega, teensy, or esp8266. the last two might require additional hardware (level shifters)

12

Def right I need a board with more memory. I will get one, but now I just want to try to understand how to access PROGMEM on principle. I come from python. Dam i had it easy.

13

This would be a simple example how to copy 3 bytes from PROGMEM to a variable in RAM. It uses memcpy_P.

In below I've declared a 3 byte array to hold a the data for a single pixel, copy 3 bytes from PROGMEM to it and print it out in the serial port.

#include <avr/pgmspace.h>

const int h = 4;
const int w = 8;
const uint8_t myArray[h][w][3] PROGMEM =
{

  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 100, 0} , {10, 20, 30}} ,
  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 100, 0} , {10, 20, 40}} ,
  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 000, 0} , {10, 20, 50}} ,
  {{0, 0, 0} , {100, 100, 100} , {200, 200, 200} , {0, 100, 100} , {100, 0, 100} , {100, 100, 0} , {100, 100, 0} , {10, 20, 60}}
};

void setup()
{
  Serial.begin(250000);

  uint8_t a;
  uint8_t b;
  uint8_t c;

  uint8_t pixeldata[3];
  
  for (a = 0; a < h; a++)
  {
    for (b = 0; b < w; b++)
    {
      // copy 3 bytes (sizeof(pixeldata))from PROGMEM to ram
      memcpy_P(&pixeldata, &(myArray[a][b]), sizeof(pixeldata));

      Serial.print("pixeldata: "); Serial.print(pixeldata[0]);
      Serial.print("/");  Serial.print(pixeldata[1]);
      Serial.print("/");  Serial.println(pixeldata[2]);
    }
    Serial.println();
  }
}

void loop()
{
}

I've modified your data slightly so one can verify that it reads correctly. The 'sizeof(pixeldata)' works correctly because pixeldata is an array of bytes; for an array of ints or longs you would need a slightly different approach.

Output:

pixeldata: 0/0/0
pixeldata: 100/100/100
pixeldata: 200/200/200
pixeldata: 0/100/100
pixeldata: 100/0/100
pixeldata: 100/100/0
pixeldata: 100/100/0
pixeldata: 10/20/30

pixeldata: 0/0/0
pixeldata: 100/100/100
pixeldata: 200/200/200
pixeldata: 0/100/100
pixeldata: 100/0/100
pixeldata: 100/100/0
pixeldata: 100/100/0
pixeldata: 10/20/40

pixeldata: 0/0/0
pixeldata: 100/100/100
pixeldata: 200/200/200
pixeldata: 0/100/100
pixeldata: 100/0/100
pixeldata: 100/100/0
pixeldata: 100/0/0
pixeldata: 10/20/50

pixeldata: 0/0/0
pixeldata: 100/100/100
pixeldata: 200/200/200
pixeldata: 0/100/100
pixeldata: 100/0/100
pixeldata: 100/100/0
pixeldata: 100/100/0
pixeldata: 10/20/60
14

Thanks!

Very kind of you to give me that example. I will attempt to get my head around it.

15

I bet if you tell us which display you're using, and which Adafruit library you're using, someone will be able to point out a function that already is set up to use PROGMEM data. The RAM shortage are Arduinos is a COMMON PROBLEM, especially when working with any sort of graphics.

16

Hi!

I'm using this display:

With this library:

I can't find many docs about this library beyond the very simple example snippets on that web page. I figured some of it out by just reading through the source, but didn't see anything relating to PROGMEM.

To be honest though, I'm probably going to have to ditch this library as it seems to completely specific to the UNO or MEGA and I want to switch boards to a NodeMCU or similar for wifi and memory reasons.

Having said that I'd still welcome any further tips anyone has with the PROGMEM stuff here.

Thanks!