Go Down

Topic: Using multidimension Arrays with Progmem  (Read 368 times) previous topic - next topic

KenK

I am trying to setup a multi dimensional array and place it in PROGMEM. 

The database in the array is defined as Unsigned char.  The required array is a 2 dimensional array.  The data in the database represents a very large graphic database.   Because of the size of the database, I need to leave it in PROGMEM.  The processor I am using is a Mega 2650 so I have plenty of program memory (256k). At this point I am only up to about 26k in Progmem.

I have setup an array system for a lot of text and it works fine in PROGMEM.  The code I am using came from the Arduino Reference Manual and looks something like this:

Code: [Select]

const char TextHdr1[]  PROGMEM = "Kloster";
const char TextHdr2[]  PROGMEM = "AutoWorks";
const char TextHdr3[]  PROGMEM = "Task List:d=Display Image x=Xoffset y=Yoffset";
const char TextHdr4[]  PROGMEM = "          W=Input Width, H=Input Height X=Input X Y=Input Y ";
const char TextHdr5[]  PROGMEM = "Display Image";
const char TextHdr6[]  PROGMEM = "Input Width (xxx)";
const char TextHdr7[]  PROGMEM = "Input Height (xxx)";


const char* const PROGMEM  TextHdrTbl[] = {TextHdr1, TextHdr2, TextHdr3, TextHdr4, TextHdr5, TextHdr6, TextHdr7};


//  To call and print the string, I am using this.
  u8g2.drawStr(8,28,strcpy_P(ScreenLabel, (char*)pgm_read_word(&(TextHdrTbl[0]))));



The section how to setup an unsigned char in the reference manual, apparently is obsolete.  I reviewed the PROGMEM.h file and it delineates the new data types but I am still not getting reliable results. 


I setup up the database in SRAM and ran the code to retrieve and display the database works fine. 


The 2 dimensional array I setup in PROGMEM is defined as follows (It will compile): 

Code: [Select]


const static unsigned char __attribute__ ((progmem))Image2dArray[160][26] ={0x00,0x00, ect};




I think the problem I am having is how to read the 2 dimensional array.  I am using the following code for that:

Code: [Select]


void DisplayImage(){
  byte i;
  byte ii;
  byte Width;
  int iii = 0;
 
  Width = width/8 -1;
  u8g2.clearBuffer(); u8g2.sendBuffer();
  for(i = Yoffset; i <= height +Yoffset; i++) {
     for(ii = Xoffset; ii <= (Width + Xoffset); ii++) {
         img[iii] = Image2dArray[i][ii];
         iii = iii+1;}
        }
 
  u8g2.drawXBM( X, Y, width, height, img);
  u8g2.sendBuffer();
}



Is it possible to put (or leave) a 2 dimensional array in PROGMEM??

How do I get the data out basically one byte at a time??

 


MorganS

An image, when it's compressed for storage, is no longer two-dimensional. A GIF is just a long series of bytes that must be read in linear order. There's no reason to store it as a 2D array. Just make it one big array.

It appears that your image is not compressed in any way, so it is possible to address a single pixel in the image and fetch that from memory. You can still do this, if you want. You just need to know how C actually implements 2D arrays.

Code: (Pseudocode) [Select]
byte fetchPixel(int x, int y) {
  return dataArrayPointer[headerLength + y*rowLength +x];
}


i is a silly name for a variable, but it's traditional. ii is even worse.
GoForSmoke: "What GShield? You never mentioned a shield."

KenK

Thanks for your reply.  

Your are correct.  The database I am using is a modified XBM.  As a general rule, the database is just a serial string of bytes.  In this case each bit in a byte represents a single pixel. (Monochrome).  

The twist I am using here is the database I have generated is a map.  The target database will be about about 500 pixels by 500 pixels.  That makes the database consume 31,250 bytes of RAM.  Now the (current) display is a 128 by 64 pixel display (for now).  What I am doing is displaying the map in 128 X 64 pixel sections.  I move around the map using the X and Y offset variables.  The X and Y offset variables will eventually be generated using a GPS card.  By making the database a 2 dimensional database, I can use the 2 "for loops" to pick out any 128 X64 pixel section on the map.  Works great in SRAM with a small database.  

As for the i & i, I use them in for loops where the variable for local use only. I know Lazy

In the past hour, I have found another clue to the problem.  

If I assign the array calls hard, I will get the data I ask for.  

  
Code: [Select]

  img = Image2dArray[0][1];
  img = Image2dArray[0][2];
  img = Image2dArray[0][2]; //ect.
 

If I try to assign the array calls using a "for loop" , I will get zeros. 

 
Code: [Select]

  for( int i = 0; i <= 2: i++) {

  img = Image2dArray[0][i];
  img = Image2dArray[0][i];
  img = Image2dArray[0][i]; //ect.
  }
 

The problem appears to be pointer related.  If you look at the way the reference manual tells you to set up string arrays, it kind of looks like PROGMEN doesn't support 2 dimensional arrays.  Bummer


Anybody got any ideas??
 


AWOL

Code: [Select]
  img = Image2dArray[0][1];
  img = Image2dArray[0][2];
  img = Image2dArray[0][2];
maybe I'm missing something here, but what do you think that does?
"Pete, it's a fool looks for logic in the chambers of the human heart." Ulysses Everett McGill.
Do not send technical questions via personal messaging - they will be ignored.

KenK

The "char" variable "img gives me the data in Image2dArray [row 0] [position x]

Whatever "X" is.


As I stated in my second post this afternoon, The program works very well if all the data is SRAM.   It is when I try to use PROGMEM that every thing falls apart.

Since my last post, I tried to get a single dimension (char) array to work.  Again, no luck.  If I call for the data using a hard coded calls, I get the data.  If I try to do it dynamically, I get junk.


Code: [Select]


const static unsigned char __attribute__ ((progmem)) Image_1d_Array[234]={
//   1     2     3     4     5     6     7     8    9     10    11    12    13   
   0x00, 0x0e, 0xf8, 0xff, 0xdf, 0x03, 0xc0, 0x1d, 0x60, 0x1f, 0x60, 0x00, 0x03,
   0xdb, 0x01, 0x00, 0x08, 0xe3, 0x07, 0x00, 0x0e, 0x00, 0x1e, 0x30, 0x00, 0x00,
   0x00, 0x0e, 0xf0, 0x07, 0xfc, 0x01, 0xc0, 0x39, 0x00, 0x1f, 0xe0, 0x00, 0x03,
  };

//This works
  Serial.print("Image= "); Serial.println(Image_1d_Array[0],HEX);
  Serial.print("Image= "); Serial.println(Image_1d_Array[1],HEX);
  Serial.print("Image= "); Serial.println(Image_1d_Array[2],HEX);
  Serial.print("Image= "); Serial.println(Image_1d_Array[3],HEX);
  Serial.println("");


//This does not
  for(i = 0; i <=11; i++){
     Serial.print("AImage= ");
     myChar = Image_1d_Array[i];
     Serial.println(myChar,HEX);
    }



The coding requirements for PROGMEM have changed and I have not been able to make heads or tails out of it.  I have to believe that the problem involves how I'm trying to access the array.

As I mentioned above, it works for strings, I cannot get it to work for simple data.

johnwasser

The compiler keeps track of the address of an array but it does not keep track of which address space it is in (SRAM, FLASH, or EEPROM). The compiler assumes SRAM so if you want to fetch from a different address space you have to use special functions.  Use those special functions!

Your test with constant indexes works because the compiler has everything it needs to determine the contents of that element at compile time.  It doesn't generate any code to fetch data so it doesn't matter which address space the array is in.
Send Bitcoin tips to: 1G2qoGwMRXx8az71DVP1E81jShxtbSh5Hp

KenK

#6
May 21, 2017, 04:25 pm Last Edit: May 21, 2017, 04:36 pm by KenK
Thanks for all of your input.  With a lot of research and a lot of trying, I got it.

It works great.  I am using a Mega 2560 processor and three 31k byte 2 dimensional arrays running in PROGMEM.  

The program is running great with 105,602 bytes of Program memory used and only 2855 bytes of SRAM used.  

All text for menus is also running in PROGMEM.  

The three Image arrays are included using the "#Include" instruction so I don't have to work around the huge arrays.  



Below is the Array assignment and the calls I use to access the arrays:  

Code: [Select]


const byte Image_2d_Array_2[500][62] PROGMEM ={
//Image2dArray[Vertical][Horizonal]{
//ElMirage_3B1 width 496 bits  height 500 bits  31000 bytes
//   1     2     3     4     5     6     7     8    9     10    11    12    13  
   0x80, 0xff, 0xff, 0xc1, 0x1c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0xe0, 0x8e, 0x03, 0x00, 0x00, 0x98, 0x01, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x63, 0x00, 0x00, 0x00, 0x0e, 0x00, 0x00, 0x00, 0x00, 0x00, ...
};


img[iii] = pgm_read_byte_near(&Image_2d_Array_1[i][ii]);





I wish the compiler would allow for bigger arrays (the max is 32765).

I hope this helps others out as there is not a lot of information on using the PROGMEM function for data arrays.  Lots on "String"Arrays.


Again, thanks to all of you for your assistance with this

Ken

Whandall

Be prepared to switch from pgm_read_byte_near() to pgm_read_byte_far() when your data reaches 64k.
Ah, this is obviously some strange usage of the word 'safe' that I wasn't previously aware of. (D.Adams)

KenK

Thanks for the tip.  
This morning I added the third Array but have not tried to access it yet.  I am surprised that even 2 arrays worked as the total program size with 2 arrays is about 75k.  When I added the third array, the program size jumped up to a little over 102 k.  

I plan on adding at least three more arrays assuming that I have room.  Once I do that, I am sure things are going to begin to get tight.  I would love to get 6 more in but I don't think that is going to happen.


Again thanks for the tip.
Ken

Whandall

You will have some fun with  pgm_get_far_address too,
because just taking the address does not work above 64k, pointers are strict 16 bit in 8 bit AVRs.

Program Space Utilities
Ah, this is obviously some strange usage of the word 'safe' that I wasn't previously aware of. (D.Adams)

KenK

When I loaded the third, database in and tried to run the program, things did not go so well.  

A simple print statement like "Serial.println("BOOT") won't work in the "Setup() section of the program.


Print statements sent to serial port 0 work.  


Most if not all of my text statements I put in PROGMEM work in the main program but not in the Setup section (I haven't tested them all yet).  


The GLCD library I am using seems to be inoperable as I can't get anything to show up on the display.  

At this point I haven't tried to read part of one of the large arrays loaded into PROGMEM and print some of the data to serial port 0 yet but that is next.


I found something else happening that really has me concerned.  I commented out sections of code for test purposes.  After recompiling the code, when I re-ran program, the commented out sections of code still worked.  HUH???  I really don't know how the compiler works, but I get the feeling that the compiler may not clear the flash RAM above 64k before reloading the new program.


Ill let you know what happens when I try that.


 :smiley-confuse:

Whandall

If you have a lot of code and data for PROGMEM, it can happen that the translation arrays
for pinMode, digitalRead and analogRead get shifted above 64k,
which will not work and create strange errors.

There is a fix for that here in the forum, but I don't have the link handy.
Ah, this is obviously some strange usage of the word 'safe' that I wasn't previously aware of. (D.Adams)

KenK

I have run some more testing on the problem.  

I tried to put a simple piece of code in to read a small section of the first array in PROGMEM.  

Code: [Select]

  for(i = 0; i <=10; i++) {
      img[i] = pgm_read_byte_far(&Image_2d_Array_1[0][i]);
      Serial.print("img= "); Serial.println(img[i],HEX);



The code works if I only have two of the 3 arrays included in PROGMEM.  As soon as I add the third array, everything blows up. 

The program behaves so badly that I could not learn anything from the test code with 3 arrays included. 

Just to insure that one of my arrays wasn't the problem, I ran the program with arrays 1&2, 2&3 and 1&3.  Again, it works with 2 but not 3. 

From this I think I can pretty much say it's a memory/compiler problem. 

As for the size of the program, it is really pretty small.  No digital I-O is being used except for the hardware SPI which drives the display.  The code that is there is mostly for setting up VARs for testing and that is done through serial port 0.  The total size of the test program with all libraries and no data is about 10k. 


In the final program, there will be a LOT more code, hardware and digital I/O that will be used.  The current code that exists less the image database currently is about 50k.

I will try to search for the post you mentioned.  I will also search the StackOverflow web site.

Have a good day and again thanks for your input.







 

AWOL

Code: [Select]
for(i = 0; i <=10; i++) {
      img[i] =
Please tell me that img has at least eleven elements
"Pete, it's a fool looks for logic in the chambers of the human heart." Ulysses Everett McGill.
Do not send technical questions via personal messaging - they will be ignored.

KenK

Actually, the Var "img" is a display buffer for a 128x64 display.  It is actually a 1024 byte array "img[1024];"

The array "Image_2d_Array_1[0]" is a 31,000 byte array "Image_2d_Array_1[500][62]".

Go Up