Best approach to move certains "lines" from a text file inside an SD card

Hi, I am doing some data logging and I will be adding data to a text file each time I cant send my data to a server.

I would like to clean the file somehow, you know because it could get too big and also take too much time to loop through because the purpose of the file is to later try and re-send the data to the server again.

My concern is: If move the data to another file and then delete the first file, if there is a power shut, I could lose a lot of information.

What is the best way you guys think so I won't lose all my info?

Since there are two main actions taking place: Delete the original file and rename the new file, a power loss could mean losing all of my stuff.

Here is my main idea:

1-Create a new file.
2-Insert desired data from the old file.
3-Return true when done.
4-Delete the old file.
(Problem here is power loss).
5-Rename the new file to the old file's filename.

As you can see, if I don't rename the new file, then the next time I try to access my data, the file won't be there.

Let's say I switch 4 and 5. Now, in order to rename the new file to the old file's filename, I first need to rename the old file, because as far as I am aware, you should not (or can't) have two files with the same exact filename.

This is the solution, so tell me if this works :

1-Create a new file.
2-Insert desired data from the old file.
3-Return true when done and store in EEPROM.
4-Rename the old file to oldFileTemp.
(Possible power failure here ?)
5-Return true when done and store in EEPROM.
6-Rename the new file to newFIle.
7-Return true when done and store in EEPROM.
8-If every step is true, then delete the old file.
9-Reset all variables to false.

Now the next time I try to access my old file to log new data, I could check if has been rename, then access it through the new name, even after a power loss.

I hope you guys understand my idea.

TL;DR: How to implement SQL Server transaction-like functions in Arduino.

I don't think it has to be that complex. At startup, if you have two files, delete the new file. Then whenever it's time to make the copy, try again. If it is a success, delete the old one.

wildbill:
I don't think it has to be that complex. At startup, if you have two files, delete the new file. Then whenever it's time to make the copy, try again. If it is a success, delete the old one.

What if there was a power loss when the new file was just created and it's either empty or just half-empty? How can you tell it should be kept or not?

If there are two files, the process failed before completion. Delete the new one and try again.

Depending on your logging frequency you could limit the amount of log entries in a file to 10 entries and name the log files with incremental numbers (or date and time stamp if possible). That way you can send the entries from the file with the lowest number when the server is available and then delete that file. Each log-entry could have a unique number which would allow the server to disregard entries already received. Doing it like this would make it unnescessary to move data from one file to another and it would be near impossible to loose data due to power loss.

Doing it like this would make it unnescessary to move data from one file to another and it would be near impossible to loose data due to power loss.

By doing this, I guess I can delete the file only if the data inside has been sent, this means that if the log file number 1 attempts to send all the data but the last line cant be sent, then everything will be kept in the file. So we will be in the same situation as we started unless we could move out the lines that have been sent but that's the same stuff from my first post. This could lead to a file growing too much or too many files in the SD card.

Could you write something on each row you have sent to indicate that it's no longer needed?

If the entries have a unique id, the server could simply ignore already received entries. Or you could replace the first character of each send entry with a "special character", that marks the entry as already sent.

const char SEND_MARKER = '*';

while (!file.atEnd())
{
  filePosStart = file.position();
  entry = file.readLine();
  if ((entry[0] != SEND_MARKER) && sendToServer(entry))
  {
    filePosEnd = file.position();
    file.setPosition(filePosStart);
    file.write(SEND_MARKER);
    file.setPosition(filePosEnd);
  }
}

I think I have found the solution to make it "ATOMIC":

First, we need to see where we accessing the files, in my case in two functions:

Note: This solution attempts to call replaceFile in a defined time, let's say each day at 23:00 and being able to keep writing new data to the file without having to "fix" the failed replace function in each boot because the file could be very large and take some time to finish the replacing and maybe you don't want this to happen.

writeData and replaceFile.

Now we are going to use 3 file names :
oldFile1
oldFile2
oldFile3

  1. The first thing to do is to move the data we want to keep from oldFile1 to a new file called oldFile2.

  2. Once we are 100% done, store a boolean in EEPROM indicating that it's all done.

  3. Then we rename oldFile1 to oldFile3.

  4. We know can rename oldFile2 to oldFile1.

  5. Now delete the oldFile3 and set the EEPROM value to false.

The example code:

void useFile(){
    if(isDataLoaded){
        if(isOldFile3Present == true && isOldFile1Present == true){
            //write to oldFile1
            //New data is stored in oldFile1 because the replace function moved the data but didnt deleted the oldFile3
        }else{
            //write to oldFile2
            //if the oldFile3 is no created yet, the new data is stored in oldFile2
            //if the oldFile3 is created but oldFile1 does not exist, this means the data is still in oldFile2 and needs to be renamed
        }
    }else{
        //write to oldFile1
        //New or old data is stored on oldFile1
        //Either replace() has not been called yet or it did finish
    }
}

void replace(){
    if(isOldFile3Present == false && isDataLoaded == true && isOldFile2Present == false){
        //set dataLoaded to false
    }
    if(isDataLoaded){
        if(isOldFile3Present){
            if(isOldFile1Present){
                //delete oldFile3 and set dataLoaded to false
            }else{
                //rename oldFile2 to oldFile1 and do the rest
            }
        }else{
            //rename oldFile1 to oldFile3 and do the rest
        }
    }else{
        //do everything again
    }
}