How to store 3D byte vectors?

Projects Programming
1

Hello,

at the moment i am building an advent calendar with an Arduino Mega. This is my first Arduino project and first time with C(++) (just Java /Python before). In the middle is a 12x12 RGB LED matrix. I want to display each day a different picture (frame) with the LEDs. For this i added a clock module, so i can get always the current day.

Each image i store in a 3D byte vector, one for each day: 1 dimension is the frame (some days have multiple frames for animation, 2. one is the channel (red, green, blue) and 3rd dimension is the Byte itself. Each bit represents one LED. here is an example:

//Note
std::vector<std::vector<std::vector<byte>>> getDec03() { 
    return {
      {
        {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000}, //R
        {B00000000,B00000000,B00000000,B00000001,B11000011,B10011100,B00111001,B11000000,B10011100,B00100001,B00000000,B10000100,B00100001,B00000000,B11111100,B00000000,B00000000,B00000000}, //G
        {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
      },
      {
        {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000}, //R
        {B00000000,B00000000,B00000000,B00000000,B00000011,B10000000,B00111001,B11000011,B10011100,B00111001,B00000000,B10000100,B00100001,B00000000,B10000100,B00111111,B00000000,B00000000}, //G
        {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
      }
  };
}

There is a helper method to return the vectors:

std::vector<std::vector<std::vector<byte>>> getFrames(uint8_t day) { 
  switch (day) {
    case 1:
      return getDec01();
    case 2:
      return getDec02();
    case 3:
      return getDec03();
...

which is called to get the frame(s) of the current day:

std::vector<std::vector<std::vector<byte>>> frames;
...
frames = getFrames(getCurrentDay());

My problem is, that when i want to compile my program with all iamges i get the following message:

Der Sketch verwendet 51292 Bytes (20%) des Programmspeicherplatzes. Das Maximum sind 253952 Bytes.
Globale Variablen verwenden 6556 Bytes (80%) des dynamischen Speichers, 1636 Bytes für lokale Variablen verbleiben. Das Maximum sind 8192 Bytes.
Wenig Arbeitsspeicher verfügbar, es können Stabilitätsprobleme auftreten.

The program won't do anything on the Arduino with this warning, it seems to crash soon after the start. The amount of Bytes in all the vectors is around 3672. I attached for some other parts an SD shield to the Arduino. My idea was to store all vectors on the SD card, maybe one file for each day. But i have no idea how to write and read the data. All examples i could find handle a lot more easier data and with my own ideas i came after over a week to end now.

Can you help me how to solve my problem? I am open to other solutions like eeprom (which i tried too, but no success). I would be happy if i could keep my frame data layout even if my be not the nicest, but it wa sa lot of work :slight_smile:

Thank you.

2

Start here... https://www.arduino.cc/en/tutorial/memory

3

Hey, thank you for your answer. i already read this page and the linked examples but it doesn't help me. For example i tried the PROGMEM keyword on my vectors, but it didn't change anything.

4

Then the problem is in the code you didn't post.

5

I changed the methods who return the frames like this:

std::vector<std::vector<std::vector<byte>>> getDec00() { 
    const PROGMEM std::vector<std::vector<std::vector<byte>>> frames = {
      {
        {B10110010,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
        {B11010100,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
        {B11101000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
      },
      {
        {B00000000,B00000000,B00000000,B00000001,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
        {B00000000,B00000000,B00000000,B00000001,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
        {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
      },
      {
        {B00000000,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
        {B00000000,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
        {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
      },
      {
        {B00000000,B00000000,B00000000,B00000100,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
        {B00000000,B00000000,B00000000,B00000100,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
        {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
      }
  };
  return frames;
}

But when i look at the used bytes in dynamic space (at least what the compiler says), there is no difference with and without progmem.

6

I suspect that std::vector is going to ignore the PROGMEM directive. The purpose of using vectors is they may change in size. Therefore they always use dynamic memory. (Even if the pointer to the vector is constant.) Your vectors aren't going to change. They're constants.

Use regular arrays.

7

And perhaps use run-length encoding or some other simple compression scheme, as your patterns seem sparse.

8 
std::vector<std::vector<std::vector<byte>>> getDec00() {
    const PROGMEM std::vector<std::vector<std::vector<byte>>> frames
.
.
.

Sorry, I don't have an answer to your problem (except to suggest you're running low on/out of memory) but I have to say to whomever is f__king around with the syntax of this language: Please, for the love of all that is holy, just stop already. It's fine as it is. When you get to the point that a declaration has twelve ':' characters you've probably gone down the rabbit hole...

9

Thank you all for answeres. Based on this i tried again to find a solution with arrays and the SD card. I ended up with two methods, one for writing my frames of the day to the SD card and for reading.

As the files need to be written on the SD card just once, there is a little manual setup for each day necessary (change the 3D vector and the day / filename). I Iterate over the first 2 outer vectors and then i write the content of the 3rd vector as byte array in a file. Because the files don't need to be written later i the main sketch, I created an extra sketch. My complete sketch for this looks like this:

#include <StandardCplusplus.h>
#include <vector>
#include <SPI.h>
#include <SD.h>


int day = 25;
std::vector<std::vector<std::vector<byte>>> frames = {
  {
     {B10110010,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
     {B11010100,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
     {B11101000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
   },
   {
     {B00000000,B00000000,B00000000,B00000001,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
     {B00000000,B00000000,B00000000,B00000001,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
     {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
   },
   {
     {B00000000,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
     {B00000000,B00000000,B00000000,B00000010,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
     {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
   },
   {
     {B00000000,B00000000,B00000000,B00000100,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //R
     {B00000000,B00000000,B00000000,B00000100,B00000000,B01000000,B00011111,B11000001,B11110000,B00001111,B10000001,B11110000,B00000111,B00000000,B11100000,B00000010,B00000000,B00000000}, //G
     {B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000} //B
   }
};

void setupSD() {
  if (!SD.begin(4))
  {
    Serial.println("sdcard initialization failed!");
    return;
  }
  Serial.println("sdcard initialization done.");
}


void writeToFile(int day) {
  String fileName = String(day) + String("dec.bin");
  byte colorArray[18];
  File myFile;
  
    myFile = SD.open(fileName,  FILE_WRITE);
    Serial.println("Opened file to write");
    Serial.println(fileName);

    for(auto& it_frames : frames) {
      for(auto& it_colors : it_frames) { //Komplettes Array einer Farbe, also 3 pro Frame
        
          std::copy(it_colors.begin(), it_colors.end(), colorArray);
          myFile.write(colorArray, sizeof(colorArray));
          
      }
    }
    Serial.println("Closing file");
    myFile.close();
}

void setup() {
  Serial.begin(9600);
  Serial.print("Initializing SD card...");
  setupSD();
  writeToFile(day);
}

void loop() {
  // put your main code here, to run repeatedly:

}

In my main sketch i created the opposite method. To know the size of the most outer vector (the number of frames for this day), I store an array on the arduino which holds this number (not very automatic, but the data should never change):

int dailyNumFrames[26] = {4,1,1,2,1,1,4,2,4,2,8,1,4,1,2,1,4,6,4,1,5,5,1,3,3,1};

I then read the previously stored array in an array and then i convert this to an 1D vector. Because the length is fix, i always know that the length is 18 bytes:

myFile.read(colorArray, sizeof(colorArray));
std::vector<byte> v(colorArray, colorArray + sizeof colorArray / sizeof colorArray[0]);

Then i add this 1D vector in a 2D vector and when i finished reading all 3 color channels, i put the 2D vector in a 3D vector. Here is the complete method:

std::vector<std::vector<std::vector<byte>>> readFromFile(int day) { 
  String fileName = String(day) + String("dec.bin");
  byte colorArray[18];
  File myFile;
  int num_frames = dailyNumFrames[day]; //Read from array, depending from day

  Serial.println("Open file to read");
  myFile = SD.open(fileName, FILE_READ);

  std::vector<std::vector<std::vector<byte>>> frames_dest;
  frames_dest.resize(num_frames);
    
  std::vector<std::vector<byte>> v_colors;
  v_colors.resize(3);
   
    for (int i = 0; i < num_frames; i++) {    
      for (int j = 0; j < 3; j++) {
      
        myFile.read(colorArray, sizeof(colorArray));
        std::vector<byte> v(colorArray, colorArray + sizeof colorArray / sizeof colorArray[0]);
           
        v_colors[j] = v;
      }
      
      frames_dest[i] = v_colors;
    }

    myFile.close();
    Serial.println("File Closed"); 
    
  return frames_dest; 
}

Might not be the nicest code, but for me it is working and maybe someone might find it helpfull.

Thanks.