Memory size blown by simply referencing a table in an include

I have a program which is close to topping out memory on an Uno.

"Global variables use 1,515 bytes (73%) of dynamic memory"

In the .h file that is being included, there is an 800 byte table. That is filling up memory but it is still OK.

But, when I reference the table in a single statement, memory usage goes up another 800+ bytes and blows through more than what is left.

Why does the simple act of referencing a table copy the entire table, making a second copy in memory?

Here is the single line, if not commented, blows memory up to more than is available:

BaseBuf[0][i] = FullASCIIBtmap8[CharLoc][i];  // The KILLER line which is fetching a single byte!

FullASCIIBtmap8 is the 800 byte table. It is already defined in a .h file. Why is referencing it blowing up memory so much? Cannot get through the compile and, or course, cannot upload to the Uno.

As an alternate, I tried:

memcpy(BaseBuf[0],FullASCIIBtmap8[CharLoc],8);

Same problem.

So, 2 questions:

Why is this happening? (Compiler bug?)

Is there any way of referencing this table (directly by address?) that will not cause a second copy of it to be made by the compiler?

OK, so that's 3 questions. So sue me! ;-))

When I plug in a Mega, the extra memory requirement is not a problem and the program runs. But it is slower than the Uno at updating the display on I2C bus. Don't want to give up the speed and would rather be able to run this on a Mini or Micro in the end.

I am not good at using addressing and memory pointers so any help with ideas and code examples appreciated. I am hoping that an address pointer to the top of, or into the table would not cause the memory explosion.

Mike Morrow

(table too large for the forum post, had to remove most of it for size)

const static byte FullASCIIBtmap8[100][8]={
//bottom   .     .     .     .     .     .    top
  {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}, // 000 Char 032 (blank space)
  {0x00, 0x18, 0x00, 0x18, 0x18, 0x3c, 0x3c, 0x18}, // 001 Char 033 (!) V3
.....
  {0xaa, 0x55, 0xaa, 0x55, 0xaa, 0x55, 0xaa, 0x55}, // 098 Char 130 (Checker 2)          V3
  {0x00, 0x08, 0x1C, 0x2A, 0x0A, 0x2A, 0x1C, 0x08}  // 099 Char 131 (U.S. Cent sign)     V3
};

Moderator edit: [u]CODE TAGS[/u]

Can you write a short sketch which shows the problem in less than 30 lines? It would make it easier to see what is going on there if you have an example which we can compile and run.

I don’t think we need the full table at this point although if you don’t mind sharing it, you can attach it to your post when using the full reply option, not the “quick reply” box.

Since you’ve declared the table as const, why not put it in PROGMEM?

Actually, I think what might be happening is that it is not including the table at all until you reference it. It just seems like twice because the rest of your program variables are about that big on their own.

BaseBuf[0] = FullASCIIBtmap8[CharLoc];

Is that the only place that the BaseBuf array is referenced ? If so, then the compiler may not be including it if it is not referenced.

Try compiling this with and without the statement assigning zero to one of the elements of the aLong array.

long aLong[500];

void setup()
{
  aLong[0] = 0;
}

void loop() {}

With the assignment

Global variables use 2,009 bytes (98%) of dynamic memory

Without the assignment

Global variables use 9 bytes (0%) of dynamic memory

Is that the only place that the BaseBuf array is referenced ? If so, then the compiler may not be including it if it is not referenced.

Right. It sounds like your memory consumption is 1500 bytes WITHOUT the 800byte table. Reference the table, and it gets included, adding an additional 800 bytes and pushing you over 2k.

There are tools that will analyze memory usage, if you're comfortable dropping down into a command-line window.

If it's a constant table, just leave it in flash (PROGMEM). Problem solved.

Do you know the difference between these two beside that actual text string contents?

Serial.print( "This string gets copied to RAM from flash at startup." );

Serial.print( F( "This string doesn't get copied to RAM, it prints directly from flash." ));

Oftentimes using the latter can cut down on RAM use to a major degree. Maybe it would leave room for your table, if you want to be able to modify it in runtime.

Sorry for delay answering. I will try to create a small demo tomorrow. It it is cool enough to work here.

The table is defined, used and referenced in the associated library. It is a bit coded character table which I have expanded to include Upper, Lower and special characters. Most, or all of the ASCII table for display on an 8x8 Matrix. I will be uploading the entire project to my web site soon as I get through this little hang. I need to access the table directly to get the bit patterns so I can implement smooth scrolling across all matrices of the 8-matrix display. This may be too slow once I get it to work. Will see...

The table is definitely referenced in the .cpp code in the library. And that is why the code is so big already (before I mention it in the code). It is already in memory and in use. But when I reference the same table in my .ino code, that's when memory gets shot.

More as soon as I can get back to it. It has been too hot here and I am keeping computers and servers turned off so they won't melt. Not a good thing! Only need air-con about 10 days a year (normally) so not cost justified but this year we have a month or more of very hot. Global warming has come here to live!

Thanks for the ideas. I will research that PROGMEM thing. Have not really read about it. Might be good but will take some thinking, I expect. Since the table is not "mine", it belongs to the library. But that's what we are here for!

Mike

Most, or all of the ASCII table for display on an 8x8 Matrix.

In the end, assuming that you use about 5 columns (bytes) per character and there are 255 characters, you will need to have 1275 bytes (or so) of RAM used by the table, leaving not much for anything else. Best to byte the bullet and move it to PROGMEM now.

MikeyMoMo: The table is definitely referenced in the .cpp code in the library. And that is why the code is so big already (before I mention it in the code). It is already in memory and in use. But when I reference the same table in my .ino code, that's when memory gets shot.

I think that you are simply underestimating the optimizing feature of the GCC compiler.

If you include a library, where a library function references the table, but you neither call and use the result of that library function in your code nor you reference the table directly and actually use the value in your sketch like sending it to Serial, the optimizer will simply optimize out the table and not include the table in the compiled sketch.

Referencing a few const values from a table that are constants at compile time may lead to the optimizing, that the compiler actually includes those few const values directly instead of including the whole table im RAM.

I'd do like MorganS suggested: Declare the table in PROGMEM and then use the pgm_read_byte() function to read out the bytes instead of addressing the table addresses directly.

jurs: I think you are simply missing the point. The table WAS included in the build (in memory). It took up 800+ bytes. I watched it grow and was worried but it fit when I was done. It started by taking up less but took more as I built it up to 100 entries. That was already running me to 85% memory usage. It WAS referenced. It WAS used. All in the library. That was mentioned specifically.

Then, when I referenced it in my sketch, it added ANOTHER 800 bytes and that was more than the UNO had to offer! That seems like a compiler bug. Why make a second copy of the table. Not sure where to submit that for consideration.

I was certain that was clearly stated when I said the table was defined, referenced and used in the library program. Obviously not clear enough.

Now the suggestion of using this modified format to start the array made a big difference:

const static byte FullASCIIBtmap8[100][8] PROGMEM = { //bottom . . . . . . top {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}, // 000 Char 032 (blank space)

Memory is no longer blown. However, there is a referencing problem because the characters shown on the display are now incorrect. I will have to figure out what adding PROGMEM messed up something elsewhere. Not sure why that would happen, but it did. Something lost track of where the data is. No errors. Program compiles and apparently uses PROGMEM but the reference to it is now wrong somehow.

So... on to finding out what is messed up by the first fix.

I always use the "F("whatever")); format for strings. No gain there but thanks for the suggestion. It did take a while to stumble across that little memory saver.

Thanks to MorganS for the PROGMEM idea. I have not come across that in the two books I have read so far. A bit deep for these intro books, I expect.

Ain't programming fun?!?!?! I will post all this somewhere when I get everything, including me, happy.

More later... Mike

Then, when I referenced it in my sketch, it added ANOTHER 800 bytes and that was more than the UNO had to offer! That seems like a compiler bug. That's not a believable scenario, and you're not showing an understanding of what we mean by "referenced." Just because the table is in a .h file, and accessed from library code, doesn't mean it will be included in the image. You have to have your sketch actually CALL the library functions that access the array from the library code. Otherwise, the compiler figures out that the library function isn't being called, and omits it, and then since there is no library function accessing the array any more, it omits the array too.

Post all your code, copies of the .elf file for both cases, OR output from "avr-nm -S --size-sort myfile.elf" (for both cases.) Otherwise, we're just guessing. (but this is extremely unlikely to be "a compiler bug", since many people are writing similar code that doesn't have the problem you are seeing.)

Once data is declared as PROGMEM, you can no longer access it directly. You have to use something like:

  BaseBuf[0][i] = pgm_read_byte(&FullASCIIBtmap8[CharLoc][i]);
[/i][/i]

Instead of

  BaseBuf[0][i] = FullASCIIBtmap8[CharLoc][i];[/i][/i]

How can I say this more clearly. I am about out of ways to say it.

I AM USING, CALLING, INCLUDING IN MY PROGRAM CODE CALLS TO THE FUNCTIONS - PROGRAMMING LANGUAGE STATEMENTS TO CALL THE ROUTINE THAT USES THIS DATA IN THE LIBRARY CODE!!! Why is that so very hard to understand. I have said it over and over.

IT IS BEING INCLUDED IN THE BYTE COUNT OF THE ASSEMBLED CODE!!! I have said this over and over.

THE CODE IS BEING CALLED THAT REFERENCES THE DATA IN THE ARRAY THAT IS DEFINED IN THE LIBRARY PROGRAM (.h file for this who don’t know). The program to display this data on a LED matrix would be useless without calling the routines that reference (read) and send this data to the display with SPI.

GOT IT???

IT *IS BEING DEFINED, USED AND REFERENCED ALREADY.
Have I said that enough now? I hope so!

Please stop feeding back to me that it is not being used. I have been programming in over a dozen languages for a lot of years. This is not my first rodeo!!! But this C code, being written strictly for classroom use is difficult to get used to. Sad that it got out of bounds and influenced so many other languages which do stupid stuff, too. Specifics off topic here.

For those of you who stood the screaming (intended), attached is a ZIP file. There is a demo program and two variations of the library code. One with PROGMEM included in the array definition and the added routine to fetch the bytes from that different area (pgm_read_byte_near). That version does NOT work properly. It runs OK but fetches the wrong data. I don’t have any idea where PROGMEM stores the array or how to find out where pgm_read_byte_near is looking to pick up the wrong data. This is getting really deep into the Atmel firmware and GCC compiler.

The other version is the one that also nearly fills memory and runs correctly.

The only two changes are in the .h file to add PROGMEM to the array definition and the last code line of the .cpp file to add “pgm_read_byte_near” to the data read. “pgm_read_byte_far” does not compile. Claims it is not defined in the scope. Documented as working but does not. Have not looked at that problem yet.

I will not pursue the data duplication compiler bug here. Without that duplication of data, none of the rest of this would be necessary and I would already be done… Bed time now. Attacking this again tomorrow.

Mike

LedControlMM8.zip (15.8 KB)

What IDE version are you using?

I have your example compiling…

Um. Where does the "BaseBuf[0] = FullASCIIBtmap8[CharLoc]*; // The KILLER line which is fetching a single byte!" go? I don’t see any basebuf at all… *
I also don’t see it including FullASCIIBtmap8 at all in the symbol dump. Or in the source. Is it called ABitmap8 now? Or is that separate?
You have:
*_ <em>*const static uint8_t ABitmap8[100][8] = {*</em> _*
In the .h file. So yes, each source module that includes the .h will get its own static copy of the array. That’s what “static” DOES, right? (Without “static”, you’d get an array in each .o file, but the linker would either be smart enough to decide that they are identical, or complain about multiple definitions.)
And this is one reason why .h files shouldn’t actually define code or data. Stick your array in the .cpp file with
“const uint8_t …”, and put an “extern const uint8_t …” in your .h file instead, and you should be OK. PROGMEM is a better solution, still. I’ll look at that in a minute…

void LedControl::displayChar8(int matrix, int charIndex) {
  for (int i=0; i<8;i++) {
      setRow(matrix,i, pgm_read_byte_near(ABitmap8[charIndex][i]));
  }
}

It needs an “&” before “ABitmap8”:

setRow(matrix,i, pgm_read_byte_near(&ABitmap8[charIndex][i]));

The AVR has multiple address spaces. The normal one is RAM, which gets treated just like any architecture.
Program memory (flash) has a separate address space. pgm_read_byte() returns a byte from the flash, given the ADDRESS of the byte in the flash memory space. Normal C compiler arithmetic works fine, most of the time, so you can do arbitrarily complex things like

   foo->bar.baz[x][y]

and have the preliminary calculations work fine (assuming that everything is in RAM, UP to that point.), and then stick the & in front to get an address and use pgm_read_byte() to read the flash at that address.

(See how much faster things go when we actually have CODE to look at?) :-)

westfw: (See how much faster things go when we actually have CODE to look at?) :-)

Amen.

I AM USING, CALLING, INCLUDING IN MY PROGRAM CODE CALLS TO THE FUNCTIONS - PROGRAMMING LANGUAGE STATEMENTS TO CALL THE ROUTINE THAT USES THIS DATA IN THE LIBRARY CODE!!!!! Why is that so very hard to understand. I have said it over and over.

IT IS BEING INCLUDED IN THE BYTE COUNT OF THE ASSEMBLED CODE!!! I have said this over and over.

THE CODE IS BEING CALLED THAT REFERENCES THE DATA IN THE ARRAY THAT IS DEFINED IN THE LIBRARY PROGRAM

Things are totally so much clearer now that you shout at us.

However I think this might clear things up for you.

You mean that the same thing happens with his RAM as with that Youtube video?

When I try to look at the video it says “This video is not available in your country”.

So the error must be country dependent.
In some countries, the RAM is only used once in a sketch.
And in some other countries it says: “This RAM is not available in your country”.
:wink:

[quote author=Nick Gammon link=msg=2406813 date=1442915164] However I think this might clear things up for you.

[/quote]

It played here. Before 6AM when I was barely up. Geez Nick, warning? Lucky I hadn't et breakfast yet.