Hi,
I work for a local agency in California and I'm building a datalogger for monitoring flowmeters for irrigation in strawberry fields.
I have a Mega, an SD breakout board, real time clock and a GSM modem (Adafruit FONA, with SIM800H chip).
The logger counts pulses from the flowmeter and stores the data every 2 minutes in a string that looks like this:
"2016-10-09 12:59:00",1511,14.48,10,50
I want to send the data through the GSM modem into an FTP server (I have this part figured out already). The logger would collect data during the day (say 6AM to 10PM) and send the data at night. I also want to keep the data in the SD card just in case the communication fails.
I'm wondering what's the best file management strategy:
1- Creating one file containing only one string every two minutes. At the end of one day, if I log from 6AM to 10PM I would end with 16 hours, 960 minutes and 480 files.
Then the sketch will have to open each file, write its line into a char array, send it to the FTP server and move the line into a master file, appending it at the end of it.
This seems complicated, but has the advantage of keeping each file, each piece of information separated.
2- Appending during the day to a single file on the SD card. At the end of the day I have one single file with 480 entries. Now I need to read each line, one after the other from where I left the pointer last time and FTP them. Then I have to append the whole file to the master file.
Any help is appreciated!
I would go with option 2 since that is similar to what I've done in a couple of my projects.
Create a new file each day with a name of the form YYYYMMDD.csv. Every 2 minutes open the file for appending, write the data and then close it.
Pete
Hi Pete,
thanks for your reply. I understand how to append data to a daily file every 2 minutes, but what happens when I need to send the data through the modem to the FTP server?
I will need to read each line from the file, make a character array and send it to the FTP server, then append the line to a backup file in the SD card.
Then I need to move to the next line in the file. How do I get back exactly where I was before? Maybe I can use position() to save the position in the file where I was during the previous iteration and get there again using seek()? Do I need to increase this number by 1 to get to the next line? Or maybe by 2 because there are two characters (/n) at the end of each line?
Finally, how to I break the loop when I get at the end of the file? "while (dataFile.available())" does the trick?
Thanks!
Gerry
Two things come to mind here - make sure when you append to the file, you open the file, append then close the file. That way, if the system goes down, you only lose the last write if it happened during the write. As far as reading from the file, I have not worked with the SD library with the Arduino, but usually, it is "while(!EOF) fd" or something along those lines to test for the end of the file as you are reading line by line.
gpsmikey:
I have not worked with the SD library
And if you DO use the SD library, you don't need that stuff, as it appends by default, and the DumpFile example included in the IDE is all you need to read it. About the only thing you can get wrong is improper formatting of the card, which applies no matter what library you use. There is a note about this at the head of this forum.
As in reply #2, I think a new file at midnight is the way to go. You already have the RTC.
void loop() {
GetClock();
if (today != day)
{
today = day;
getFileName();
}
.
.
void getFileName(){
sprintf(filename, "%02d%02d%02d.csv", year, month, day);
}
is one approach.
Hi all,
thanks for your replies.
Nick, thanks, I'm familiar with DumpFile. I'm somewhat confused because at line 50 it says " // if the file is available, write to it:" , while I think it should say "read from it".
But it appears from your reply that indeed the command
while (dataFile.available()) will break out of the while when (and only when) I get to the end of the file.
I also found the function readStringUntil() and I'm using it to make a string of a line from the file.
while (myFile.available()) {
MyStr = myFile.readStringUntil('\n');
I've been unable to read the line into a char array instead of into a string. Is there a way to do that?
Thanks!
Gerry
gerardospinelli:
at line 50 it says " // if the file is available, write to it:" , while I think it should say "read from it".
I didn't notice that, but I guess you are right.
I've been unable to read the line into a char array instead of into a string. Is there a way to do that?
I'm afraid I can't comment - other than to question the need.... My data is collected as numbers, stored on SD as timestamped numbers, is read from SD as timestamped numbers, is Serial.written as timestamped numbers via bluetooth, for ultimate reading and use as numbers. I can't see the point of strings or chars for this.
On a slightly different tack, you can use dtostrf to derive a string from a float. I use this to get a graphic display on the phone. I don't see the point in it for data collection, and only do it because this particular graph software demands it.
void StringOut()
{
dtostrf(InTemp,4, 2, strIn);
dtostrf(OutTemp,4, 2, strOut);
dtostrf(Rise,4, 2, strRise);
dtostrf(Fall,4, 2, strFall);
str4 = str1 + strIn + str2 + strOut + str2 + strRise + str3;
Serial2.println(str4);
}
Float InTemp is made into string strIn etc. I'm not at all sure this approach is of any use to you.
Hi Nick,
thank you for your reply.
The reason I want to read the string into a char array is that the modem uses some AT command that require what I'm sending to be a char array.
I'm currently reading from the file to a string using readStringUntil('\n'); an then turning the string into an array with:
dataString.toCharArray(data,str_len);
but maybe there is a more efficient function to read from the file directly to a char array.
Thanks!
gerardospinelli:
The reason I want to read the string into a char array is that the modem uses some AT command t.....................
but maybe there is a more efficient function
Possibly, but it seems that you might have justified your position!
Great, thanks.
I have another issue.
Everyday I make a file adding a line every two minutes. At night I read one line at the time from the file, and send the data to the FTP server one line at the time. When I get to the end of the file I want to remove that file and append its content to a backup file in the SD card, in a folder called backup.
I have three solutions I can think of (other suggestions are welcome!):
1- I simply append each line to the backup file after I FTPed it. When I get to the end of the daily file, I will have to delete it. This is risky because if for some reason the program breaks out of the while (dataFile.available()) loop, I delete data that I havent FTPed nor backuped
2- Just FTP each line and when I get to the end of the daily file, append its whole content to the backup file. I haven't been able to find a "move" function. It seems that I would have to read the whole content into dynamic memory, append it to the backup file and then delete the original file. My file should have a little less than 48000 characters. I think that that correspond to 48kBytes so I can't fit into a Mega's 8kB SRAM. Is this correct?
3- Just leave each file on the SD card, the name of each file tells me the date it was created on. The issue here is that at night when I send the file, I look for the one with today's date.
Despite maybe being the simplest, this is my least preferred solution, because I end up with my backup data in the SD card scattered into daily data and I would have to put them all together if the FTP fails.
Thanks everybody for your help. I'll send you some strawberries when we're done with this project! 
Gerry
I'm not sure I understand this, but it seems a pointless exercise in file manipulation.
1- I simply append each line to the backup file after I FTPed it. When I get to the end of the daily file, I will have to delete it. This is risky because if for some reason the program breaks out of the while (dataFile.available()) loop, I delete data that I havent FTPed nor backuped
I don't understand this. It can't be more than a matter of sequence but, if there is a problem, you can fix it by not deleting anything.
3- Just leave each file on the SD card, the name of each file tells me the date it was created on. The issue here is that at night when I send the file, I look for the one with today's date.
That is essentially what I do, but I don't see it as an issue. I just send the date, and get the file. I have only lately realised this is not ideal, but that is because of the nature of the data, not the file operation. I am now working on having two files, one with the daily data, and a yearly with one line per day of summaries. It seems that you could do something similar, and for the similar reasons.