Hi, I have created a data logger that should run non stop.
As I was not aware of memory related problem resulting from the use of Strings I have used them all over the code.
The code is now 1200+ lines and runs flawlessly on simulations.
The project is about to be put into service and after reading about situations where it runs fine for days or weeks the system may crash due to memory fragmentation and related problems I got concerned that it may happen with my project as well.
So far I have tried it several times for days and it runs ok.
When compilation finishes Arduino IDE shows the following message:
The sketch uses 46798 bytes (18%) of space. Max = 253952 bytes.
Global variables = 3229 bytes (39%), living 4963 bytes to local variables, Max = 8192 bytes.
Now the question directed to experienced people on this matter:
Should I:
Use one of the several methods found at the internet to reboot Arduino once a day, say 00:00hs would be fine, to start over with a clean ordered memory?
Will it work?
Invest time on removing all uses of String objects and replace it with arrays of strings?
The second option obviously seem to be the most time consuming of the two and I still have to learn how to do it reliably.
Another related question:
I am storing sensor data to SD thus using the File object and the project also outputs data to Bluetooth thus sing another object.
Will the use of those do as much damage to memory as Strings?
In that scenario, will avoiding Strings altogether make my project stable (not considering other flaws it might have)?
The idea that you realize there's likely a problem, but offering to put some chewing gum and bailing wire on it by a restart seems to be a poor choice. If fragmentation is occurring, how do you know the code will play nice for 24 hours? The String class can have the fragmentation problem you mention, but part of that depends upon how you write the code. The class also bloats the code size in most cases.
The real solution is to avoid the fragmentation problem, not tiptoe around it. Using the C string char array approach is probably a better long-run solution. Take some time playing with the C string functions summarized here and I think you'll find it's easier to master them than trying to play Chicken with the String class.
It is not a good idea to use the String (capital S) class on an Arduino as it can cause memory corruption in the small memory on an Arduino. This can happen after the program has been running perfectly for some time. Just use cstrings - char arrays terminated with '\0' (NULL).
When using Cstrings you must use strcmp() to compare values rather than ==
If it works effectively for the purpose of avoiding memory collapse I do not really need Strings in most of my code.
Those at the top are pieces of code I copied from the internet to compose my solution.
OK, I did not post more of the code because it is too big.
But this function should give you a better view as the entire code is 1300 lines long.
There are no global Strings, only local.
void reIndex() {
String line;
File indexFile;
float jDonfile, jD = 0;
unsigned long recPosition, recSize;
unsigned long startProc = 0;
//erase existing index.txt
if (SD.exists("index.txt")) {
SD.remove("index.txt");
}
Serial3.println("Indexação vai levar vários minutos.");
Serial3.print("Indexando: ");
Serial3.println(filename);
startProc = millis();
if (SD.exists(filename)) {
dataFile = SD.open(filename, O_READ);
if (dataFile.available()) {
line = dataFile.readStringUntil('\n');
recSize = dataFile.position(); //determina tamanho do registro
dataFile.seek(0);
// set date time callback function
SdFile::dateTimeCallback(dateTime);
indexFile = SD.open("index.txt", O_RDWR | O_CREAT);
while (dataFile.available()) {
line = dataFile.readStringUntil('\n');
jDonfile = line.substring(0, 10).toFloat();
recPosition = dataFile.position() - recSize;
if (jD != jDonfile) {
Serial3.print(".");
indexFile.print(jDonfile);
indexFile.print(":");
//Add trailing zeros
if (recPosition < 1000000000000) {indexFile.print("0");}
if (recPosition < 100000000000) {indexFile.print("0");}
if (recPosition < 10000000000) {indexFile.print("0");}
if (recPosition < 1000000000) {indexFile.print("0");}
if (recPosition < 100000000) {indexFile.print("0");}
if (recPosition < 10000000) {indexFile.print("0");}
if (recPosition < 1000000) {indexFile.print("0");}
if (recPosition < 100000) {indexFile.print("0");}
if (recPosition < 10000) {indexFile.print("0");}
if (recPosition < 1000) {indexFile.print("0");}
if (recPosition < 100) {indexFile.print("0");}
if (recPosition < 10) {indexFile.print("0");}
indexFile.print(recPosition);
indexFile.println("");
indexFile.flush();
jD = jDonfile;
}
}
indexFile.close();
dataFile.close();
}
}
Serial3.println("");
Serial3.println("Fim da indexação.");
elapsedTime(startProc);
}
That is what I am trying to find out, if the change I made avoiding to declare a String will not do it means changes in the code will have to be larger than I anticipated.
As I have to use SD and it comes with methods that "look like" Strings than by just using the SD file system am I using Strings?
No the SD filesystem does not require Strings. But the way you are using it does. Operations like .substring() create a temporary String and it is all the temporaries that are fragmenting your memory.
I have some pretty important Arduino projects that use Strings and have no problems. It may be your pattern of usage which is causing fragmentation. I think if you make your strings global then they won't be created and destroyed as often so the fragmentation will be better.
You create a new String. the previous String cannot be used because no one knows how long the previous one was. So, the memory used keeps growing and growing.
That is a great information about declaring a String Global, most of them are just as in my code, used as a way to get sub-strings.
I am not having problems at this point.
The Data-logger works for days (in simulations) without a flaw but I am trying to make it more robust even before putting it to work.
Is there a way to know when fragmentation is taking place?
Dear Paul_KD7HB, yes, thanks, I am starting to understand that.
Looks like the suggestion to work with a Global String may do the trick with minimum impact.
Thanks for your time.
The global String advice is not a good one IMHO (unless you initialize it with a big big size)... memory is still allocated dynamically at object instanciation or modification and substring operations create other. Worse if the next byte after your global one has been used by a small one (which will happen when you do String operations) and if you need to expand the global one then the whole memory gets moved - that’s how you poke holes...
you could play around with the c_str() method which is a direct pointer to the c-string buffer (a null terminated char array) to see if it is moving around as your code executes (printing the buffer address of all your Strings along the loops and seeing things stay at the same place would be a good indication - even if not enough as there are invisible Strings created for you)
jremington:
No need to do that, because it IS taking place.
No you can’t scientifically guarantee it’s always the case.
If allocation and deallocation are LIFO then the code is fine
The only challenge is that operations such as + are creating invisible Strings and you can’t control exactly what happens in memory, it’s hidden in the class.
Thus agree with the conclusion that it’s better to get rid of the class
It's a hack (using Serial.print in the String class does not make sense but it was the quickest way for me to write some demo code); don't forget to add a Serial.begin in setup.
i am facing same mayhem like you do, built a product using sim 800L , mega 2560 , rfid , it was way more simpler using Strings than strings, eventhough exhausted ram never go beyond 35% in a day , i set a timer ( using at+cclk on sim 800L ) to reboot the system daily at 02:00 hours and it does work pretty cool