Using multidimension Arrays with Progmem

KenK · May 19, 2017, 8:30pm

I am trying to setup a multi dimensional array and place it in PROGMEM.

The database in the array is defined as Unsigned char. The required array is a 2 dimensional array. The data in the database represents a very large graphic database. Because of the size of the database, I need to leave it in PROGMEM. The processor I am using is a Mega 2650 so I have plenty of program memory (256k). At this point I am only up to about 26k in Progmem.

I have setup an array system for a lot of text and it works fine in PROGMEM. The code I am using came from the Arduino Reference Manual and looks something like this:

const char TextHdr1[]  PROGMEM = "Kloster";
const char TextHdr2[]  PROGMEM = "AutoWorks";
const char TextHdr3[]  PROGMEM = "Task List:d=Display Image x=Xoffset y=Yoffset";
const char TextHdr4[]  PROGMEM = "          W=Input Width, H=Input Height X=Input X Y=Input Y ";
const char TextHdr5[]  PROGMEM = "Display Image";
const char TextHdr6[]  PROGMEM = "Input Width (xxx)";
const char TextHdr7[]  PROGMEM = "Input Height (xxx)";


const char* const PROGMEM  TextHdrTbl[] = {TextHdr1, TextHdr2, TextHdr3, TextHdr4, TextHdr5, TextHdr6, TextHdr7};


//  To call and print the string, I am using this.
  u8g2.drawStr(8,28,strcpy_P(ScreenLabel, (char*)pgm_read_word(&(TextHdrTbl[0]))));

The section how to setup an unsigned char in the reference manual, apparently is obsolete. I reviewed the PROGMEM.h file and it delineates the new data types but I am still not getting reliable results.

I setup up the database in SRAM and ran the code to retrieve and display the database works fine.

The 2 dimensional array I setup in PROGMEM is defined as follows (It will compile):

const static unsigned char __attribute__ ((progmem))Image2dArray[160][26] ={0x00,0x00, ect};

I think the problem I am having is how to read the 2 dimensional array. I am using the following code for that:

void DisplayImage(){
  byte i; 
  byte ii; 
  byte Width;
  int iii = 0; 
  
  Width = width/8 -1;
  u8g2.clearBuffer(); u8g2.sendBuffer();
  for(i = Yoffset; i <= height +Yoffset; i++) {
     for(ii = Xoffset; ii <= (Width + Xoffset); ii++) {
         img[iii] = Image2dArray[i][ii]; 
         iii = iii+1;}
        }
  
  u8g2.drawXBM( X, Y, width, height, img);
  u8g2.sendBuffer();
}

Is it possible to put (or leave) a 2 dimensional array in PROGMEM??

How do I get the data out basically one byte at a time??

MorganS · May 19, 2017, 8:49pm

An image, when it's compressed for storage, is no longer two-dimensional. A GIF is just a long series of bytes that must be read in linear order. There's no reason to store it as a 2D array. Just make it one big array.

It appears that your image is not compressed in any way, so it is possible to address a single pixel in the image and fetch that from memory. You can still do this, if you want. You just need to know how C actually implements 2D arrays.

byte fetchPixel(int x, int y) {
  return dataArrayPointer[headerLength + y*rowLength +x];
}

i is a silly name for a variable, but it's traditional. ii is even worse.

KenK · May 19, 2017, 9:47pm

Thanks for your reply.

Your are correct. The database I am using is a modified XBM. As a general rule, the database is just a serial string of bytes. In this case each bit in a byte represents a single pixel. (Monochrome).

The twist I am using here is the database I have generated is a map. The target database will be about about 500 pixels by 500 pixels. That makes the database consume 31,250 bytes of RAM. Now the (current) display is a 128 by 64 pixel display (for now). What I am doing is displaying the map in 128 X 64 pixel sections. I move around the map using the X and Y offset variables. The X and Y offset variables will eventually be generated using a GPS card. By making the database a 2 dimensional database, I can use the 2 "for loops" to pick out any 128 X64 pixel section on the map. Works great in SRAM with a small database.

As for the i & i, I use them in for loops where the variable for local use only. I know Lazy

In the past hour, I have found another clue to the problem.

If I assign the array calls hard, I will get the data I ask for.

  img = Image2dArray[0][1];
  img = Image2dArray[0][2];
  img = Image2dArray[0][2]; //ect.

If I try to assign the array calls using a "for loop" , I will get zeros.

  for( int i = 0; i <= 2: i++) {

  img = Image2dArray[0][i];
  img = Image2dArray[0][i];
  img = Image2dArray[0][i]; //ect.
  }

The problem appears to be pointer related. If you look at the way the reference manual tells you to set up string arrays, it kind of looks like PROGMEN doesn't support 2 dimensional arrays. Bummer

Anybody got any ideas??

system · May 19, 2017, 9:50pm

  img = Image2dArray[0][1];
  img = Image2dArray[0][2];
  img = Image2dArray[0][2];

maybe I'm missing something here, but what do you think that does?

KenK · May 19, 2017, 11:08pm

The "char" variable "img gives me the data in Image2dArray [row 0] [position x]

Whatever "X" is.

As I stated in my second post this afternoon, The program works very well if all the data is SRAM. It is when I try to use PROGMEM that every thing falls apart.

Since my last post, I tried to get a single dimension (char) array to work. Again, no luck. If I call for the data using a hard coded calls, I get the data. If I try to do it dynamically, I get junk.

const static unsigned char __attribute__ ((progmem)) Image_1d_Array[234]={
//   1     2     3     4     5     6     7     8    9     10    11    12    13   
   0x00, 0x0e, 0xf8, 0xff, 0xdf, 0x03, 0xc0, 0x1d, 0x60, 0x1f, 0x60, 0x00, 0x03,
   0xdb, 0x01, 0x00, 0x08, 0xe3, 0x07, 0x00, 0x0e, 0x00, 0x1e, 0x30, 0x00, 0x00,
   0x00, 0x0e, 0xf0, 0x07, 0xfc, 0x01, 0xc0, 0x39, 0x00, 0x1f, 0xe0, 0x00, 0x03,
  };

//This works
  Serial.print("Image= "); Serial.println(Image_1d_Array[0],HEX);
  Serial.print("Image= "); Serial.println(Image_1d_Array[1],HEX);
  Serial.print("Image= "); Serial.println(Image_1d_Array[2],HEX);
  Serial.print("Image= "); Serial.println(Image_1d_Array[3],HEX);
  Serial.println("");


//This does not
  for(i = 0; i <=11; i++){
     Serial.print("AImage= ");
     myChar = Image_1d_Array[i];
     Serial.println(myChar,HEX);
    }

The coding requirements for PROGMEM have changed and I have not been able to make heads or tails out of it. I have to believe that the problem involves how I'm trying to access the array.

As I mentioned above, it works for strings, I cannot get it to work for simple data.

johnwasser · May 20, 2017, 10:50pm

The compiler keeps track of the address of an array but it does not keep track of which address space it is in (SRAM, FLASH, or EEPROM). The compiler assumes SRAM so if you want to fetch from a different address space you have to use special functions. Use those special functions!

Your test with constant indexes works because the compiler has everything it needs to determine the contents of that element at compile time. It doesn't generate any code to fetch data so it doesn't matter which address space the array is in.

KenK · May 21, 2017, 2:25pm

Thanks for all of your input. With a lot of research and a lot of trying, I got it.

It works great. I am using a Mega 2560 processor and three 31k byte 2 dimensional arrays running in PROGMEM.

The program is running great with 105,602 bytes of Program memory used and only 2855 bytes of SRAM used.

All text for menus is also running in PROGMEM.

The three Image arrays are included using the "#Include" instruction so I don't have to work around the huge arrays.

Below is the Array assignment and the calls I use to access the arrays:

const byte Image_2d_Array_2[500][62] PROGMEM ={
//Image2dArray[Vertical][Horizonal]{
//ElMirage_3B1 width 496 bits  height 500 bits  31000 bytes
//   1     2     3     4     5     6     7     8    9     10    11    12    13   
   0x80, 0xff, 0xff, 0xc1, 0x1c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0xe0, 0x8e, 0x03, 0x00, 0x00, 0x98, 0x01, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x63, 0x00, 0x00, 0x00, 0x0e, 0x00, 0x00, 0x00, 0x00, 0x00, ...
};


img[iii] = pgm_read_byte_near(&Image_2d_Array_1[i][ii]);

I wish the compiler would allow for bigger arrays (the max is 32765).

I hope this helps others out as there is not a lot of information on using the PROGMEM function for data arrays. Lots on "String"Arrays.

Again, thanks to all of you for your assistance with this

Ken

Whandall · May 21, 2017, 3:06pm

Be prepared to switch from pgm_read_byte_near() to pgm_read_byte_far() when your data reaches 64k.

KenK · May 21, 2017, 9:46pm

Thanks for the tip.
This morning I added the third Array but have not tried to access it yet. I am surprised that even 2 arrays worked as the total program size with 2 arrays is about 75k. When I added the third array, the program size jumped up to a little over 102 k.

I plan on adding at least three more arrays assuming that I have room. Once I do that, I am sure things are going to begin to get tight. I would love to get 6 more in but I don't think that is going to happen.

Again thanks for the tip.
Ken

Whandall · May 21, 2017, 9:57pm

You will have some fun with pgm_get_far_address too,
because just taking the address does not work above 64k, pointers are strict 16 bit in 8 bit AVRs.

Program Space Utilities

KenK · May 22, 2017, 1:43pm

When I loaded the third, database in and tried to run the program, things did not go so well.

A simple print statement like "Serial.println("BOOT") won't work in the "Setup() section of the program.

Print statements sent to serial port 0 work.

Most if not all of my text statements I put in PROGMEM work in the main program but not in the Setup section (I haven't tested them all yet).

The GLCD library I am using seems to be inoperable as I can't get anything to show up on the display.

At this point I haven't tried to read part of one of the large arrays loaded into PROGMEM and print some of the data to serial port 0 yet but that is next.

I found something else happening that really has me concerned. I commented out sections of code for test purposes. After recompiling the code, when I re-ran program, the commented out sections of code still worked. HUH??? I really don't know how the compiler works, but I get the feeling that the compiler may not clear the flash RAM above 64k before reloading the new program.

Ill let you know what happens when I try that.

Whandall · May 22, 2017, 1:48pm

If you have a lot of code and data for PROGMEM, it can happen that the translation arrays
for pinMode, digitalRead and analogRead get shifted above 64k,
which will not work and create strange errors.

There is a fix for that here in the forum, but I don't have the link handy.

KenK · May 22, 2017, 3:29pm

I have run some more testing on the problem.

I tried to put a simple piece of code in to read a small section of the first array in PROGMEM.

  for(i = 0; i <=10; i++) {
      img[i] = pgm_read_byte_far(&Image_2d_Array_1[0][i]);
      Serial.print("img= "); Serial.println(img[i],HEX);

The code works if I only have two of the 3 arrays included in PROGMEM. As soon as I add the third array, everything blows up.

The program behaves so badly that I could not learn anything from the test code with 3 arrays included.

Just to insure that one of my arrays wasn't the problem, I ran the program with arrays 1&2, 2&3 and 1&3. Again, it works with 2 but not 3.

From this I think I can pretty much say it's a memory/compiler problem.

As for the size of the program, it is really pretty small. No digital I-O is being used except for the hardware SPI which drives the display. The code that is there is mostly for setting up VARs for testing and that is done through serial port 0. The total size of the test program with all libraries and no data is about 10k.

In the final program, there will be a LOT more code, hardware and digital I/O that will be used. The current code that exists less the image database currently is about 50k.

I will try to search for the post you mentioned. I will also search the StackOverflow web site.

Have a good day and again thanks for your input.

system · May 22, 2017, 3:31pm

for(i = 0; i <=10; i++) {
      img[i] =

Please tell me that img has at least eleven elements

KenK · May 22, 2017, 9:03pm

Actually, the Var "img" is a display buffer for a 128x64 display. It is actually a 1024 byte array "img[1024];"

The array "Image_2d_Array_1[0]" is a 31,000 byte array "Image_2d_Array_1[500][62]".

system · May 22, 2017, 9:05pm

I was just wondering why, if you wanted to iterate eleven times, you went through the mental gymnastics of including a ten in your loop condition.

Whandall · May 22, 2017, 9:17pm

KenK:
I have run some more testing on the problem.

I tried to put a simple piece of code in to read a small section of the first array in PROGMEM.
  for(i = 0; i <=10; i++) {

img[i] = pgm_read_byte_far(&Image_2d_Array_1[0][i]);
Serial.print("img= "); Serial.println(img[i],HEX);

pgm_read_byte_far(&Image_2d_Array_1[0][ i]); Oops. :o

This can only work below 64k, as I already mentioned.

Whandall:
You will have some fun with pgm_get_far_address too,
because just taking the address does not work above 64k, pointers are strict 16 bit in 8 bit AVRs.

Program Space Utilities

Any address used for the _far variants has to originate from a pgm_get_far_address 'call'.

An expression &whatEver only delivers the lower 16 bits of the address.

Whandall · May 22, 2017, 9:24pm

KenK · May 22, 2017, 9:33pm

Whandall:

I have spent most of the day looking for and studying posts concerning the 64k issue. I found a very interesting and complicated post that took place back in 2013:

"Just a newby asking the 64k question again - Arduino Mega2560 - Programming Questions - Arduino Forum"

I covers the issue quite well. The guy asking the questions, is trying to use arrays that are 64 in size. That in itself is a problem but it present an interesting discussion.

One of the problems is the way the compiler (Linker) puts things together. It appears that the linker sets up the memory (Flash) such that "Text" is placed at the bottom followed by "PROGMEM" and then by the "Program Functions". This causes a problems because most libraries do not expect to be placed above 64k. This includes the actual Arduino hardware. This explains why when I add the third array, things like "Serial.print" and my display fails to function properly.

It seems to me that the easiest way to get around this problem would be to instruct the "Linker" to put the "PROGMEM" data starting at 64K and leave the bottom 64k for the actual program functions. That would work for me but it would limit the program size to 64k. The best way to do it would be to put the program functions on the bottom and place the Text and PROGMEM on top of based on how much room was required by the program functions.

Either way, any of this is over my head.

Whandall · May 22, 2017, 10:02pm

Yes, when the tables for digitalRead and the other pin mappings leave the first 64k,
the pgm_read_byte_near of the functions fails and maps pins into nirvana.
Any code using the abstract layer will then fail miserably.

The Just a newby asking the 64k question thread seems to be more relalted
to cases where the size of code or data is more than 64k.

Topic		Replies	Views
Accessing an array in PROGMEM Programming	16	1173	September 30, 2023
Problem beim Auslesen von langen Arrays in PROGMEM Deutsch	8	56	December 28, 2025
PROGMEM Phänomen Deutsch	57	282	January 16, 2026
MEGA + TFT + progmem Programming	21	133	October 18, 2025
Problem uploadig over 100 kb to mega2560 board General Guidance	26	3559	May 5, 2021

Using multidimension Arrays with Progmem

Related topics