Creating a folder named according timestamp and writing files to it

Hi guys,

I currently have 2 projects which, individually, are running great:

  • GPS: Arduino Nano running a u-blox NEO-6M-v2 Sensor to collect GPS data
  • SD Cardreader: Same Arduino Nano creating and writing to files and directorys

I have begun merging the 2 Projects to get a board that will collect the GPS data and send it to the SD card. So far so good, this works perfectly......

Soooo......what is my problem? Everything is working great? Weeeellll......the way i have it set up, i can only startup and shutdown the programm once. If i restart, all previosly collected data is lost, and i start out all over again. (I could continue writing to the "old" file, but i want individual files for each "session").

My idea is: I want to create a folder accoording to Timestamp "YYYYMMDD-HHMM", and write my file "GPS.txt" into that folder. Upon my next session, i want it to create a new folder with the new time, and write the data to a new "GPS.txt".

So far so good, but i am running into various Problems:

  • Timestamp: The SD card sets up the file structure in the Setup. The first data (GPS RTC) is collected by the GPS only in the loop function, so late for the creation. I could put the file creation into the function, but i only need to perform it once.....if i put it in an if-clause to check if it has been created i need to check on each and every cycle, which is ... well...time wasted. Is there a more efficient way?
  • Name creation: Right now i get YYYY, MM, DD and so on as individual data packages. I need to combine them into the complete package to create it. Not only that, but i also need to create another package being called "YYYYMMMDD-HHMM/GPS.txt" so i can adress and read the file. What format would be most useful for this? String?

Any hints or ideas would be very welcome.
The nano is coming pretty much to its limits, im running 70% program memory, and over 80% dynamic memory. I am hoping this will clear up some as soon as i remove all the serial debugging "serial.print", but still its very tight....

Update Problem 1:

if (Created==0){
  sprintf(filename, "%04d%02d%02d-%02d02d",  gps.date.year(),gps.date.month(), gps.date.day(),gps.time.hour(),gps.time.minute());
  Serial.println("");
  Serial.println("");
  Serial.println("");
  Serial.println("");
  Serial.println("");
  Serial.print("F I L E N A M E : ");
  Serial.println(filename);
Created=1; 
}

filename was declared globally as "String" and as "char", both do not work. Instead they make the programm hang up....

  1. Are you restricted to DOS style file names (8.3 format)? if so, your file name is too long for the file system.
  2. I hope you meant that filename was an 'array of char' and not 'char'. Overwriting the end of a character buffer is often a good way to crash your program.
  1. I personally am not :slight_smile: But im guessing arduino is. I was hoping the folder name is not so limited...

  2. Originally i did nothing. That did not compile because "filename" was not declared. I tried declaring it in multiple formats, none of them worked......

You might consider putting the date in the file name, Instead of that stupid GPS.txt, thereby avoiding the need for a folder.

That is my backup idea.....however then i will be restricted to 8+3 and my declaration will not work.
Plus, in the final application, i will be wanting 2-3 file types. One raw data, one a marked-up google earth file, and one excel type cvs with all values (speed, heading etc....later on potentially other sensor data).
Most elegant would be a folder for each data package....

Also possible: folder "YYYYMMDD" and file "HHMM.txt".

Thats just markup.....my main problem is still basic creation.

Quick question: in my posted code, how do i have to declare "filename"?
String filename; does not work.....

I have no idea what you mean by your declaration not working. I understand you can use longer filenames with SDFat.Lib

char filename[] = "00000000.CSV";
File myFile;
void getFileName(){
sprintf(filename, "%04u%02u%02u.csv", year, month, monthDay);
}

Nick, thanks for the reply. I think it got me partially further. Here is what i have now:

char folder [15];
char earth [24];
char sens [25];

 if (sys==1){
        sprintf(folder, "%04d%02d%02d",
        year, month, day);
        sprintf(earth, "%04d%02d%02d/%02d%02d.mkv",
        year, month, day, hour, minute);
        sprintf(sens, "%04d%02d%02d/%02d%02d.csv",
        year, month, day, hour, minute);
        createFiles();
        
      }


void createFiles(){
 Serial.println("");
 Serial.println("file creation"); 
Serial.println(folder);
Serial.println(earth);
Serial.println(sens);
Serial.println(sys);
 
 SD.mkdir(folder);
 myFile=SD.open(earth, FILE_WRITE); 
 myFile.println("Test of MKV");
 myFile.close();
 myFile=SD.open(sens, FILE_WRITE); 
 myFile.println("Test of MKV");
 myFile.close();
 Serial.println("Created!"); 
 sys=2;
}

What am i trying? I am trying to create a folder with 8 or less characters (date-stamp), and 2 files within that carry the time-stamp.
I use the sprintf to fill them. (sys is a status byte. Status 0 means no GPS avail, 1 means GPS avail-> files can be created, and 2 means GPS avail and files all created.
So far it works great. BUT!!!

SD.mkdir(folder); makes the program hang up (compiles fine, stops when called). If i coment it out, the program runs through fine, but no files are created (card remains empty....)

Where is my mistake?

PS This is what is put out over serial monitor when mkdir is commented out....:

file creation
20200121
20200121/1924.mkv
20200121/1924.csv
1
folder done!
earth done!
Created!
467.71.6
20200121
20200121/1924.mkv
20200121/1924.csv
2

Found the mistake! The card did not initialize. Apparently arduino is very sensitive to having the card removed under power.....my bad.

Now having problems saving the data to the card, but will look at that tomorrow....

I am trying to create a folder with 8 or less characters (date-stamp), and 2 files within that carry the time-stamp.

That sounds reasonable as far as Arduino is concerned, but may be unreasonable as far as the real user is concerned. You may end up with a swag of unsortable files if/when you lift them out of the folder.

Miniflyer:
Apparently arduino is very sensitive to having the card removed under power.....my bad.

I guess that makes sense. It sounds like another good reason for leaving the card in there, and getting the data out via bluetooth.

Nick_Pyner:
That sounds reasonable as far as Arduino is concerned, but may be unreasonable as far as the real user is concerned. You may end up with a swag of unsortable files if/when you lift them out of the folder.

Do you have a better idea? I`m very open to suggestions. Right now im limited do 8+3. Apparently SDFat is more compact and supports long file names, but i have yet to learn how to adress within that library.

Nick_Pyner:
I guess that makes sense. It sounds like another good reason for leaving the card in there, and getting the data out via bluetooth.

Now THAT will be a problem. Mainly because of my next Problem:

Right now i am using 68% program memory, and 70% dynamic memory.
I would like to put in an xml file header into one of the files......i have the full data of what is to be written. Its about 18 lines and a total of 450 characters.
If i add all of them with myfile.println("XML header blablabla"");, then these additional 18 lines add 18% (!!!!!!!) to my dynamic memory. With 88% i get a warning that it might cause stability problems. In fact just that happens, i cannot run the program anymore without it hanging up.

Is there a way to write to the SD card without filling up my dynamic memory?? I know theres a way for the serial.print function......but also for the sd card writing?

Miniflyer:
Do you have a better idea? I`m very open to suggestions. Right now im limited do 8+3. Apparently SDFat is more compact and supports long file names

I don't think SDFat is more compact, I understand SD is a subset. I have never used SDFat and I suspect it is more trouble than it is worth, but it might be what you need. I just make a new file at midnight and download them via Bluetooth.
I think you should consider how the data is used, not stored. You may find that your various groups of data can be sorted out in Excel.

cannot run the program anymore without it hanging up.

Sounds like time to move to a bigger MCU, like one of the new Nanos or a Pro Mega

I know theres a way for the serial.print function......but also for the sd card writing?

I don't know about this. I understand MyFile.print and Serial.print work the same way and simply send the same stuff to different destinations. This is not to say I haven't run out of memory, but I stood no chance of fixing it without getting a Mega.

I found this:

https://forum.arduino.cc/index.php?topic=428015.0

Serial.print(F("Hello World")) seems to be the syntax. I will try it. Right now i have a totally different problem, but i do not want to discuss it here (has nothing to do with topic).

In fact topic is solved....how can i declare the thread solved?

Edit the headline. It is not strictly necessary to do this. Somebody else may pick it up later.