All memory used up by 2D array in for loop

I have a 2D short array, storing the addresses of various char strings stored on EEPROM.
The data was in a row/column arrangement before loading to EEPROM and storing the addresses...

At a certain point, I would want to recall all values from a row...
before I even got to that point, I wanted to print the full row values to to ensure I stored on EEPROM correctly... so I wrote this small function to emulate closely how it would work.

short itemAddress[100][5];

void setup() {
  test()
}

void loop() {}

void test() {
  short row = 20;
  for (short i=itemAddress[row][0] ; i < itemAddress[row][4] ; i++) {
    Serial.write(EEPROM.read(i));
  }
}

Ignore the fact there is nothing else... the script is massive and this one function is the issue.
I am unable to verify this function as it runs out of memory... It uses 111% (2287 bytes) of dynamic memory.

Compare this to:

short itemAddress[100][5];

void setup() {
  test()
}

void loop() {}

void test() {
  for (short i=0 ; i < 1000 ; i++) {
    Serial.write(EEPROM.read(i));
  }
}

This change makes my entire script use 62% (1287 bytes) of dynamic memory...
The former should only output 4 char string items but fails, and the latter will happily print out ALL data on EEPROM...

Firstly: Why does the former use SO much memory when it should only be accessing 2 numbers from a 2D array?

Secondly: Apart from scrapping the whole idea and reading the data on the fly rather than storing on EEPROM... is there a way I can use my stored addresses to access the data I need?

Your second sketch doesn't use the array, so the compiler happily optimises it away.

1 Like

Where are you initializing the values in itemAddress? The for statement does not make much sense, you are initializing i to a value in itemAddress, exiting the for loop when i reaches a value in itemAddress, and incrementing i by one each time, implying that the contents of itemAddress are just sequential numbers. Ok, I see where it will print out each character in every char string except for the last one in the row.

Are all char strings the same length? You could easily calculate the EEPROM address if that is the case. Otherwise, depending on the processor being used you could store the itemAddress array in program memory, or store it on the EEPROM along with the char strings.

Fair of you to ask where... it happens in a previous function (not listed because its huge and convoluting)...

But for context... the CSV data looks something like this:

Gold 9k,10c Coin,,\n
Lead,Screw,Bottle Top,6 Pence\n

So there are 4 possible items in each row, with 100 rows.
The previous function reads each char in the CSV and loads to EEPROM sequentially, while storing the starting address for each item in the "itemAddress" array, as well as the last address so I know the length of the last item...

So for this first row...
itemAddress[0][0] == 0
itemAddress[0][1] == 7
itemAddress[0][2] == 15
itemAddress[0][3] == nothing has been put here
itemAddress[0][4] == nothing has been put here

After those addressees have been saved into the array, with the data on EEPROM, is when i then call the functions in my initial post.

I hope that provides some clarity and provokes ideas where i can better handle the situation

Are all char strings the same length?

I wish... would make it a lot easier. See above post for example

store the itemAddress array in program memory

Unsure how to go about storing into program memory instead of the default dynamic memory, although insight to this might help because I'm struggling with memory optimisation for the whole project.

or store it on the EEPROM along with the char strings.

Don't know how to then access this data and how to store it in an easily retrievable way.

Lastly... as I mentioned in the initial post... I can scrap this EEPROM idea and read it from the SD card, however

  • Subsequent program startups will take longer, selecting settings to choose the correct database instead of having the option to 'load previous database' (nice to have feature but not necessary)
  • I will need to access a particular row of data dozens of time in any one session... causing dozens of SD file openings, reading, and closing. (Will that cause memory leak and eventually lock up??)

Sounds like the data can dynamically change, so storing in program memory is not a viable option.

Do you have plenty of EEPROM to work with? You could minimize the size of the itemAddress array by storing the entire csv line in EEPROM and separating the individual items when needed, that way you only need one entry per line. Storing itemAddress itself on EEPROM should not be a problem, since every element of the array is the same size a simple calculation would allow you to retrieve any desired element of the array.

Sort of. There are dynamic options to select and narrow down the data... but after its selected and loaded, it doesn't change.

Not really much EEPROM tbh... I'm probably making it more of a hassle than it needs to be. IF the data ever reached its maximum, it would be 600% of the available EEPROM storage, however I am currently only using about 30%, and realistically, don't see it going past 60%.

I'll play around with the idea you've given me though...
To clarify, i write my data to the EEPROM, then write that itemAddress array... when the user inputs a row, i then calculate which address to pull from EEPROM, read that, and then use it pull the actual text?

Sounds correct. To make things simpler, I would put the itemAddress array at the beginning or end of EEPROM, so that it always starts at a fixed address and has a fixed number of elements sufficient for the largest number of text item you will ever use.

Not for me. After I added #include <EEPROM.h>, put a ';' after test() and initialized Serial it compiled without error and takes up only 184 bytes (8%) of dynamic memory on an UNO. Of course the compiler is smart enough to knows that every element of itemAddress[][]contains zero so it doesn't really need to store the array.

Adding a function to fill the array with random values brings the memory usage up to 1188, about what you would expect with the addition of 1000 bytes of array. Still not close to 2287.

#include <EEPROM.h>

short itemAddress[100][5];

void setup()
{
  Serial.begin(115200);
  delay(200);

  fill();

  test();
}

void loop() {}

void fill()
{
  for (short row = 0; row < 100; row++)
    for (short col = 0; col < 5; col++)
    {
      itemAddress[row][col] = random(65536);
    }
}

void test()
{
  for (short row = 0; row < 100; row++)
    for (short i = itemAddress[row][0] ; i < itemAddress[row][4] ; i++)
    {
      Serial.write(EEPROM.read(i));
    }
}

Thanks for testing it and looking into it.
I did mention that I omitted the rest of my sketch as it is 400 lines long.
That by itself uses about 62% of my 2kb memory.

Your findings however do make sense..
I was able to fill the array with no issues at all to the memory, until I tried to access the values within the for loop.
That made me believe that they array wasn't taking up the 1000 bytes like you discovered and hence was confused

As you have discovered, the compiler will detect when you write to a variable but never access the data, allowing the variable to be complete eliminated when optimizing the code.