Removing Lines from SD File

Hi, I'm creating a logger and have data from a sensor going to a file on an SD card. It works great except the file keeps growing which I don't want.

How do I remove the last lines from the file and keep it at 336 records? It should grow until it hit 336 records than the first record would be deleted and a new one added and so on.

I tried adding an ID to each line so I can find the last line but nothing I do works. I'm surprised there isn't much info on removing or finding data.

This is how the data looks on the SD card.

[grW1=153.50-1]
[grW1=152.40-2]
[grW1=152.95-3]
[grW1=153.50-4]
[grW1=152.95-5]
[grW1=152.95-6]
[grW1=152.95-7]
[grW1=152.95-8]
[grW1=152.95-9]
[grW1=152.40-10]
[grW1=152.40-11]
[grW1=153.50-12]
[grW1=152.95-13]
[grW1=152.40-14]
[grW1=152.40-15]
[grW1=152.95-16]
[grW1=152.40-17]
[grW1=152.40-18]

The grW1 is a variable name that each value goes into when I load the file, this works good.

The number after the - is the ID. When ID 337 gets added ID 1 should be removed. When ID 338 gets added ID 2 should be removed, etc...

This is the code for writing the data.

void writeGraph() 
{
  // Delete the old One
  // SD.remove("settings.txt");
  // Create new one
  myFile = sd.open("graph.txt", FILE_WRITE);
  // writing in the file works just like regular print()/println() function
  myFile.print("[");
  myFile.print("grW1=");
  myFile.print(grW1,2);
  myFile.print("-");
  myFile.print(grWid);
  myFile.println("]");
  
  // close the file:
  myFile.close();
}

This is the code for reading the data.

void readGraph()
{ 
  char character;
  String settingName;
  String settingValue;
  String grHistID;
  myFile = sd.open("graph.txt");
  if (myFile) 
  {
    while (myFile.available()) 
    {
      character = myFile.read();
      while((myFile.available()) && (character != '['))
      {
        character = myFile.read();
      }
      character = myFile.read();
      while((myFile.available()) && (character != '='))
      {
        settingName = settingName + character;
        character = myFile.read();
      }
      character = myFile.read();
      while((myFile.available()) && (character != '-'))
      {
        settingValue = settingValue + character;
     //   settingName = settingName + character;
        character = myFile.read();
      }
      character = myFile.read();
      while((myFile.available()) && (character != ']'))
      {
        grHistID = grHistID + character;
        character = myFile.read();
      }
      if(character == ']')
      {        
        // Apply the value to the parameter
        applyGraph(settingName,settingValue);
        // Reset Strings
        settingName = "";
        settingValue = "";
      }
      
    }
  // close the file:
  myFile.close();
  } 
  else 
  {
    // if the file didn't open, print an error:
    Serial.println("error opening settings.txt");
  }
}
// converting string to Float
float toFloat(String settingValue)
{
  char floatbuf[settingValue.length()+1];
  settingValue.toCharArray(floatbuf, sizeof(floatbuf));
  float f = atof(floatbuf);
  return f;
}
void applyGraph(String settingName, String settingValue) 
{
  if(settingName == "grW1") {grW1=toFloat(settingValue);}
}

Any help would be appreciated, thanks.

except the file keeps growing which I don't want.

What do you want?

How do I remove the last lines from the file and keep it at 336 records?

Why would you want to remove from the end of the file, keeping the oldest data plus one new record? Keeping the 336 newest record might make sense.

The number after the - is the ID. When ID 337 gets added ID 1 should be removed. When ID 338 gets added ID 2 should be removed, etc...

So, it is NOT the last few lines you want to remove. It is the FIRST few lines.

Well, there isn't any way to do that. What you need to do is copy the data that you want to keep from the old file to a new file, add the new data, close the new file, delete the old file, and rename the new file.

Hi, I want to keep the file at 336 records, never more. I want to keep the newest 336 records and dump the older ones.

robsworld78:
Hi, I want to keep the file at 336 records, never more. I want to keep the newest 336 records and dump the older ones.

Permission granted.

From the sample data, it appears that just one number and an index change.

There is room in the Arduino RAM memory to store an array of 336 numbers. Keep that updated, and write the entire array out every once in a while to the SD card for permanent storage.

PaulS:
Permission granted.

????

jremington:
From the sample data, it appears that just one number and an index change.

There is room in the Arduino RAM memory to store an array of 336 numbers. Keep that updated, and write the entire array out every once in a while to the SD card for permanent storage.

Hi, it is just one number, the index is only for trying to find it on the card. I'm working on a big project and the Due is almost full.

Try this sketch. The sketch and two text files are included, move them to your SD card.

RemoveLinesFromFile.zip (2.44 KB)

PaulS:
Permission granted.

robsworld78:
????

PaulS gave you the steps at the end of reply #1. Implement them and you should be able to do what you want.

HazardsMind:
Try this sketch. The sketch and two text files are included, move them to your SD card.

Right on thanks! This does exactly what I want, well done!

that seems like a lot of code.

if all you want is to keep log files small, typical application logger does is to create a new file when the current file reach a certain size or number of lines, and just keep the last N files.

As to technique for finding data, if you make all your lines the same length (pad with space if needed), then to find a certain line would be a simple factor of the line length and you can file seek directly to that file position.

I have an application that keeps 1 file per day (24 hour period), and I can find the record for any time very quickly using this technique.

HazardsMind:
Try this sketch. The sketch and two text files are included, move them to your SD card.

The program is running ok not until I run it 4 consecutive times on the serial monitor.
After a few run unwanted character appears and save it on the text file. I don’t know how to fix it. Thank you.