Prevent Arduino Mega from crashing due to memory related problems with Strings

Hi, I have created a data logger that should run non stop.

As I was not aware of memory related problem resulting from the use of Strings I have used them all over the code.

The code is now 1200+ lines and runs flawlessly on simulations.

The project is about to be put into service and after reading about situations where it runs fine for days or weeks the system may crash due to memory fragmentation and related problems I got concerned that it may happen with my project as well.

So far I have tried it several times for days and it runs ok.

When compilation finishes Arduino IDE shows the following message:

The sketch uses 46798 bytes (18%) of space. Max = 253952 bytes.
Global variables = 3229 bytes (39%), living 4963 bytes to local variables, Max = 8192 bytes.

Now the question directed to experienced people on this matter:

Should I:
Use one of the several methods found at the internet to reboot Arduino once a day, say 00:00hs would be fine, to start over with a clean ordered memory?
Will it work?

Invest time on removing all uses of String objects and replace it with arrays of strings?

The second option obviously seem to be the most time consuming of the two and I still have to learn how to do it reliably.

Another related question:
I am storing sensor data to SD thus using the File object and the project also outputs data to Bluetooth thus sing another object.
Will the use of those do as much damage to memory as Strings?

In that scenario, will avoiding Strings altogether make my project stable (not considering other flaws it might have)?

Thanks in advance.
Paulo

The idea that you realize there's likely a problem, but offering to put some chewing gum and bailing wire on it by a restart seems to be a poor choice. If fragmentation is occurring, how do you know the code will play nice for 24 hours? The String class can have the fragmentation problem you mention, but part of that depends upon how you write the code. The class also bloats the code size in most cases.

The real solution is to avoid the fragmentation problem, not tiptoe around it. Using the C string char array approach is probably a better long-run solution. Take some time playing with the C string functions summarized here and I think you'll find it's easier to master them than trying to play Chicken with the String class.

It is not a good idea to use the String (capital S) class on an Arduino as it can cause memory corruption in the small memory on an Arduino. This can happen after the program has been running perfectly for some time. Just use cstrings - char arrays terminated with '\0' (NULL).

When using Cstrings you must use strcmp() to compare values rather than ==

...R

Hi, thanks for the guidance.

I have evaluated my code and it used to look like this:

      String line;
      indexFile = SD.open("index.txt", O_RDWR | O_CREAT);
      while (dataFile.available()) {
        line = dataFile.readStringUntil('\n');
        jDonfile  = line.substring(0, 10).toFloat();

And I have replaced it for:

      //String line;
      indexFile = SD.open("index.txt", O_RDWR | O_CREAT);
      while (dataFile.available()) {
        //line = dataFile.readStringUntil('\n');
        jDonfile  = dataFile.readStringUntil('\n').substring(0, 10).toFloat();

If it works effectively for the purpose of avoiding memory collapse I do not really need Strings in most of my code.
Those at the top are pieces of code I copied from the internet to compose my solution.

Thanks again
Paulo

Without seeing how your variables are defined it is impossible to know whether you have really changed anything

the line

dataFile.readStringUntil('\n').substring(0, 10).toFloat();

seems to me to be using the String class (and not because of the readStringUntil())

If you are using the String class it may work fine for 33.3 hours and then bite you in the ass.

…R

OK, I did not post more of the code because it is too big.

But this function should give you a better view as the entire code is 1300 lines long.
There are no global Strings, only local.

void reIndex() {
  String line;
  File indexFile;
  float jDonfile, jD = 0;
  unsigned long recPosition, recSize;
  unsigned long startProc = 0;

  //erase existing index.txt
  if (SD.exists("index.txt")) {
    SD.remove("index.txt");
  }

  Serial3.println("Indexação vai levar vários minutos.");
  Serial3.print("Indexando: ");
  Serial3.println(filename);
  startProc = millis();

  if (SD.exists(filename)) {
    dataFile = SD.open(filename, O_READ);
    if (dataFile.available()) {
      line = dataFile.readStringUntil('\n');
      recSize = dataFile.position();  //determina tamanho do registro
      dataFile.seek(0);

      // set date time callback function
      SdFile::dateTimeCallback(dateTime);

      indexFile = SD.open("index.txt", O_RDWR | O_CREAT);
      while (dataFile.available()) {
        line = dataFile.readStringUntil('\n');
        jDonfile  = line.substring(0, 10).toFloat();
        recPosition = dataFile.position() - recSize;

        if (jD != jDonfile) {
          Serial3.print(".");
          indexFile.print(jDonfile);
          indexFile.print(":");
          //Add trailing zeros
          if (recPosition < 1000000000000) {indexFile.print("0");}
          if (recPosition <  100000000000) {indexFile.print("0");}
          if (recPosition <   10000000000) {indexFile.print("0");}
          if (recPosition <    1000000000) {indexFile.print("0");}
          if (recPosition <     100000000) {indexFile.print("0");}
          if (recPosition <      10000000) {indexFile.print("0");}
          if (recPosition <       1000000) {indexFile.print("0");}
          if (recPosition <        100000) {indexFile.print("0");}
          if (recPosition <         10000) {indexFile.print("0");}
          if (recPosition <          1000) {indexFile.print("0");}
          if (recPosition <           100) {indexFile.print("0");}
          if (recPosition <            10) {indexFile.print("0");}
          indexFile.print(recPosition);
          indexFile.println("");
          indexFile.flush();
          jD = jDonfile;
        }
      }
      indexFile.close();
      dataFile.close();
    }
  }
  Serial3.println("");
  Serial3.println("Fim da indexação.");
  elapsedTime(startProc);
}

That is what I am trying to find out, if the change I made avoiding to declare a String will not do it means changes in the code will have to be larger than I anticipated.

As I have to use SD and it comes with methods that “look like” Strings than by just using the SD file system am I using Strings?

Thanks
Paulo

No the SD filesystem does not require Strings. But the way you are using it does. Operations like .substring() create a temporary String and it is all the temporaries that are fragmenting your memory.

I have some pretty important Arduino projects that use Strings and have no problems. It may be your pattern of usage which is causing fragmentation. I think if you make your strings global then they won’t be created and destroyed as often so the fragmentation will be better.

Every time you do this:

void reIndex() {
String line;
File indexFile;

You create a new String. the previous String cannot be used because no one knows how long the previous one was. So, the memory used keeps growing and growing.

Paul

Dear MorganS, thanks very much for your help.

That is a great information about declaring a String Global, most of them are just as in my code, used as a way to get sub-strings.

I am not having problems at this point.
The Data-logger works for days (in simulations) without a flaw but I am trying to make it more robust even before putting it to work.

Is there a way to know when fragmentation is taking place?

Thanks
Paulo

Dear Paul_KD7HB, yes, thanks, I am starting to understand that.
Looks like the suggestion to work with a Global String may do the trick with minimum impact.
Thanks for your time.

Paulo

The global String advice is not a good one IMHO (unless you initialize it with a big big size)... memory is still allocated dynamically at object instanciation or modification and substring operations create other. Worse if the next byte after your global one has been used by a small one (which will happen when you do String operations) and if you need to expand the global one then the whole memory gets moved - that’s how you poke holes...

don’t go there, just get rid of them...

Thanks J-M-L, for the tip, and how do I define the size of I string when I define it?

Is there a way to check if fragmentation is taking place?

Thanks again
Paulo

See the reserve() method

you could play around with the c_str() method which is a direct pointer to the c-string buffer (a null terminated char array) to see if it is moving around as your code executes (printing the buffer address of all your Strings along the loops and seeing things stay at the same place would be a good indication - even if not enough as there are invisible Strings created for you)

Is there a way to check if fragmentation is taking place?

No need to do that, because it IS taking place. The question is simply: how long will your code run until the Arduino crashes?

Bite the bullet and rewrite all the code to use C-strings instead.

jremington:
No need to do that, because it IS taking place.

No you can’t scientifically guarantee it’s always the case.

If allocation and deallocation are LIFO then the code is fine

The only challenge is that operations such as + are creating invisible Strings and you can’t control exactly what happens in memory, it’s hidden in the class.

Thus agree with the conclusion that it’s better to get rid of the class

pcborges:
Is there a way to know when fragmentation is taking place?

http://forum.arduino.cc/index.php?topic=539995.msg3680822#msg3680822 might be of help if you want to analyse what's happening under the hood.

It's a hack (using Serial.print in the String class does not make sense but it was the quickest way for me to write some demo code); don't forget to add a Serial.begin in setup.

Nice hack. Way too much detail for the current question.

I like to use a free memory printout to see if the memory is being consumed over time.

Dear jremington, thanks.

But knowing the time required for that to take place would give me time to reboot for instance and still using the system while the code is updated.

So, if I had a way to know how bad the system will perform would give me predictability.

Regards
Paulo

J-M-L, what do you mean by:

“operations such as + are creating invisible Strings”

Please clarify.

Thanks
Paulo

i am facing same mayhem like you do, built a product using sim 800L , mega 2560 , rfid , it was way more simpler using Strings than strings, eventhough exhausted ram never go beyond 35% in a day , i set a timer ( using at+cclk on sim 800L ) to reboot the system daily at 02:00 hours and it does work pretty cool