Reading a text file using PROGMEM

hbtousa · January 18, 2021, 7:28pm

I have a file that holds data for eight channels; each channel value is either 255 or Zero. The data shown below depicts four rows; each row has the individual value for each channel.
...
255 255 255 255 0 0 0 0
255 255 255 255 0 0 0 0
255 255 255 255 0 0 0 0
255 255 255 255 0 0 0 255
...
The timing is important. Each row should be read during a 5 milliseconds period.
Since this is file is very long, I been thinking on using PROGMEM to load the data by chunks into FLASH memory instead of reading the data off an SD Card.
As far as testing, I just ordered an SD Card, so I don't have means to use it.
I feel very confident reading off the sd card and parsing. Never before touched PROGMEM
PROGMEM looks very challenging and not sure if using it will help me at all.
Can you please, advise me what way to go? I am not sure if I am on the right path for this coding.
Arduino UNO and Nano are my boards

Thanks so much

david_2018 · January 18, 2021, 7:58pm

hbtousa:
I have a file that holds data for eight channels; each channel value is either 255 or Zero. The data shown below depicts four rows; each row has the individual value for each channel.
...
255 255 255 255 0 0 0 0
255 255 255 255 0 0 0 0
255 255 255 255 0 0 0 0
255 255 255 255 0 0 0 255
...

You could compress the eight channels into a single byte, since there are only two values, have a 1 bit represent 255 and a 0 bit represent 0.

wildbill · January 18, 2021, 8:23pm

How long is the file?

You will not be able to write on progmem at runtime.

hbtousa · January 18, 2021, 8:26pm

david_2018:
You could compress the eight channels into a single byte, since there are only two values, have a 1 bit represent 255 and a 0 bit represent 0.

Thanks for your input. Definitely, I will consider this for sure.
..
11110000
11110000
11110000
11110001
...

hbtousa · January 18, 2021, 8:32pm

wildbill:
How long is the file? Aprox 300000 bytes. With david_2018's suggestion I can lower that number really down. I also might run a query to find rows that are duplicated and compress the data further. Just an idea.

You will not be able to write on progmem at runtime.==> I was reading about it. I think I have to preload the data from the begining.

My main concern is the timing aspect.

Thanks

anon57585045 · January 18, 2021, 9:01pm

My main concern is the timing aspect.

5 milliseconds is almost an eternity in CPU time. If that is some objection to bit packing, you are way off base.

johnwasser · January 19, 2021, 4:24pm

Looks like your lines are about 17 to 33 characters each. If you have about equal quantities of '255' and '0' you would average about 25 characters per line. A 300,000-character file would hold about 12,000 lines. That would compress into 12k bytes in FLASH/PROGMEM. The Arduino UNO has 32k bytes of FLASH so you should have plenty of room.

So every 5 milliseconds you want to take a 'line' (byte) from the 'file' (array) and... do what with it?

hbtousa · January 20, 2021, 2:41pm

johnwasser:
Looks like your lines are about 17 to 33 characters each. ==>Yes, but why do you have an extra character?

If you have about equal quantities of '255' and '0' you would average about 25 characters per line. ==> Yes, you are right.
A 300,000-character file would hold about 12,000 lines. That would compress into 12k bytes in FLASH/PROGMEM.==> This is part of my problem, which I'm digging into it right now, never worked with FLASH/PROGMEM before.

The Arduino UNO has 32k bytes of FLASH so you should have plenty of room.==>If I have the room is excellent; however, I need to verify that reading off flash memory is fast enough to allow me to reading each 'line' (byte) every 5 milliseconds.

So every 5 milliseconds you want to take a 'line' (byte) from the 'file' (array) and... do what with it?===> Each channel will be attached to Arduino analog ports .

Thanks very much

hbtousa · January 21, 2021, 6:47am

const uint16_t NUMBER_OF_ELEMENTS = 16373;
const int MAX_SIZE = 32;

const char descriptions [NUMBER_OF_ELEMENTS] [MAX_SIZE] PROGMEM = {
...........
  {"255 255 255 000 000 000 000 000"},
  {"255 255 255 000 000 000 000 000"},
  {"255 255 255 000 000 000 000 000"},
  {"255 255 255 000 000 000 000 000"},
  {"255 255 255 000 000 000 000 000"},
  {"255 255 255 000 000 000 000 000"},
  {"255 255 255 000 000 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 255 255 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
  {"000 000 000 000 000 000 000 000"},
};


// Print a string from Program Memory directly to save RAM
void printProgStr (const char * str)
{
  char c;
  if (!str)
    return;
  while ((c = pgm_read_byte(str++)))
    Serial.print (c);
} // end of printProgStr

void setup ()
{
  Serial.begin (115200);

  for (int i = 0; i < NUMBER_OF_ELEMENTS; i++)
  {
    printProgStr ((const char *) &descriptions [i]);
    Serial.println ();  // finish line off
  }

}  // end of setup

void loop () {
  // end of for loop


}

After running the code I got the errors showed below.
What's wrong with the code?
Is the file too big to be handled? I am using part of the file for this test
How can I compress the data ? I haven't tried the suggestion made by
david_2018

on this thread.
Please, any input would be highly appreciate it.
Thanks

MemoryBench:5:57: error: size of array 'descriptions' is too large
const char descriptions [NUMBER_OF_ELEMENTS] [MAX_SIZE] PROGMEM = {
^
C:\Users\Owner\Documents\Arduino\MemoryBench\MemoryBench.ino: In function 'void setup()':
MemoryBench:16414:35: error: 'descriptions' was not declared in this scope
printProgStr ((const char *) &descriptions );

^~~~~~~~~~~~*
exit status 1
size of array 'descriptions' is too large

johnwasser · January 21, 2021, 6:40pm

hbtousa:
What's wrong with the code?

It says "size of array 'descriptions' is too large". If you multiply 16373 times 32 you get 523,936 or about half a megabyte. That is not going to fit in 32k.

hbtousa:
How can I compress the data ?

Start with a text editor. Change each '{"' to '0b'. Change ech '"},' to ' ,'. Change each '255 ' to '1' and each '000 ' to '0'. You will end up with something like:

const char descriptions [NUMBER_OF_ELEMENTS] PROGMEM = {
...........
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
};

The data is now 1/32nd the size.

hbtousa · January 21, 2021, 8:47pm

johnwasser:
It says "size of array 'descriptions' is too large". If you multiply 16373 times 32 you get 523,936 or about half a megabyte. That is not going to fit in 32k.
Start with a text editor. Change each '{"' to '0b'. Change ech '"},' to ' ,'. Change each '255 ' to '1' and each '000 ' to '0'. You will end up with something like:
const char descriptions [NUMBER_OF_ELEMENTS] PROGMEM = {

...........
0b11100000,
0b11100000,
0b11100000,
0b11100000,
0b11100000,
0b11100000,
0b11100000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00011000,
0b00000000,
0b00000000,
0b00000000,
0b00000000,
0b00000000,
0b00000000,
0b00000000,
0b00000000,
};




The data is now 1/32nd the size.

I managed to compress the data. I used 8536 lines and they loaded fine. I tried the code below but I haven't been able to properly retreat the data from program memory. I have tried different sketches without luck.. Can you please and again point me on the right direction?
Thanks very much

const uint16_t NUMBER_OF_ELEMENTS = 8536;

const int MAX_SIZE = 11;
const PROGMEM uint16_t descriptions []    PROGMEM = {
...  0b00110000,
  0b00110000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
};int k = 0;
unsigned int displayInt;
void setup ()
{
  Serial.begin (9600);
  Serial.println ();
    // read back a 2-byte int
  for (byte k = 0; k < 16; k++) {
    displayInt = pgm_read_word_near(descriptions + k);
    Serial.println(displayInt);
  }
  Serial.println();


}  // end of setup

void loop () {

 // displayInt = pgm_read_word_near(descriptions + k);
 // Serial.print(k);  Serial.print(" ");
 // Serial.print(displayInt);Serial.println();
//  k = (k + 1) % 8;

  
}

johnwasser · January 21, 2021, 11:39pm

const uint16_t NUMBER_OF_ELEMENTS = 8536;
const byte descriptions[NUMBER_OF_ELEMENTS] PROGMEM =
{
  0b00110000,
  0b00110000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
  0b11000000,
}; 


void setup ()
{
  Serial.begin (115200);
  Serial.println ();
  // Read and display the first 16 bytes
  for (unsigned k = 0; k < 16; k++)
  {
    byte displayByte = pgm_read_byte_near(descriptions + k);
    Serial.println(displayByte, BIN);
  }
  Serial.println();


}  // end of setup


void loop () {}

Output:

david_2018 · January 22, 2021, 12:12am

You will need to convert the compressed data back to its original value.

const byte MAX_SIZE = 32;

const byte descriptions[] PROGMEM = {
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b11100000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00011000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000,
  0b00000000
};

const uint16_t NUMBER_OF_ELEMENTS = sizeof(descriptions) / sizeof(descriptions[0]);

void printProgStr (const byte str)
{
  byte number;
  for (int i = 7; i >= 0; i--) {
    if (bitRead(str, i) == 0)
    {
      number = 0;
    }
    else
    {
      number = 255;
    }
    Serial.print(number);
    Serial.print(' ');
  } // end of printProgStr
}

void setup ()
{
  Serial.begin (115200);

  for (byte i = 0; i < NUMBER_OF_ELEMENTS; i++)
  {
    printProgStr (pgm_read_byte(&descriptions[i]));
    Serial.println ();  // finish line off
  }

}  // end of setup

void loop () {
  // end of for loop


}

hbtousa · January 23, 2021, 1:57am

I was able to successfully load in progmem around 30000 lines using that specific binary format 0b1111111.
const uint16_t NUMBER_OF_ELEMENTS = 29900;
const byte descriptions[NUMBER_OF_ELEMENTS] PROGMEM =

avrdude: writing flash (31494 bytes):

Writing | ################################################## | 100% 5.15s

avrdude: 31494 bytes of flash written.

Thanks very much for your invaluable help. The learning experience was admirable and gratifying.

system · May 23, 2021, 1:57am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
PROGMEM Phänomen Deutsch	57	278	January 16, 2026
MEGA + TFT + progmem Programming	21	132	October 18, 2025
Accessing an array in PROGMEM Programming	16	1166	September 30, 2023
Lee mal los datos si usas PROGMEM [SOLUCIONADO] Software	3	767	May 7, 2021
Question about PROGMEM (strcpy_P usage) Programming	75	856	January 18, 2026

Reading a text file using PROGMEM

Related topics