How to delete a line in a CSV file with SDFat?

Just like to title says, I want to keep a keep a log of readings that is only 300 readings long for 12+ sensors. I would like to delete the first line after 300 readings so to keep a rolling log, meaning the log will always be of the last 300 readings.

Is there a way to delete the first line of a CSV file?

It might be a lot less tedious to start a new file every 300 readings.

To delete a line you need to copy the bits you want to keep to a new file. No easy way to just do one line.

Think about a file based on number of lines, as suggested or one file per day, with the date as a name for the file.

if the lines are of equal length you can reset the file pointer and overwrite the file on that position.
(should be possible, with sdfat lib, open in RW mode)

How do you want to process those 300 lines?
Because depending on your needs there may be some other issues.
e.g. you need to add a time stamp to each line to be able to sort them.

If I start a new file then there is a very good chance that I will not get the 300 readings I want.

The 300 readings are 5 minutes worth of 1 second interval readings. So I want to be able to review and graph trends with the data in the CSV file on small lcd. I can't just let the CSV just gather the whole days worth of data because I don't think it will be readable on the graph.

The easiest thing I could think of the create a rolling set of data, is to delete the first making all of the others index down and add a new 300th readings.

Not being able to do this really puts the brakes on my project. Is there another method of storing close to 3600 pieces of data that can be easily recalled, rewritten or erased?

because I don't think it will be readable on the graph.

You aren't going to graph anything on the Arduino, are you? Let the system that is doing the graphing, and that has boatloads of memory, deal with the stuff that takes a lot of memory.

RobDrizzle:
The 300 readings are 5 minutes worth of 1 second interval readings. So I want to be able to review and graph trends with the data in the CSV file on small lcd. I can't just let the CSV just gather the whole days worth of data because I don't think it will be readable on the graph.

You're probably right about that.

It might pay to give up on the "small LCD" and export the data to Excel using PLX-DAQ. You can even watch the graph form live and, trust me, plotting a graph on a large LCD beats the hell out of plotting on a small one driven by an Arduino. If you are on station you can set the start for a 300 line window, or you can gather a swag of data and grab the 300 lines that best suits at your convenience. Note that Excel is only good for 65000 rows of data, but you might find that sufficient.

PaulS:

because I don't think it will be readable on the graph.

You aren't going to graph anything on the Arduino, are you? Let the system that is doing the graphing, and that has boatloads of memory, deal with the stuff that takes a lot of memory.

Yeah I was... I didn't think a function that would write 300 pixels would be that demanding. The graphing function is something that I would like to do on the fly without having to use a computer. The sensors are install in the field where we typically don't have access to a computer.

If I got anSPI memory IC and then just wrote each of the sensors data as a 300 element array to it, would that work?

Yeah I was.

Then, ignore everything else I said.

If you could make your records fixed length (not that difficult), the read time to read a file with 300 records, and use the last 300, would be the same as the read time, and use the last 300 records, for a file with 57000 records. This is because you only need to read the last 300 records, regardless of how many there are.

With variable length records, of course this is not true, since you actually need to read each character/byte to determine how many records there are, and then you need to read the file again, one byte/char at a time, to skip the first n-300 records.

Then the way to think about the SD file is not as a file but an offline array of values, with a fixed record size. This has already been suggested.

I would use a combination of a circular buffer (size n=300 records) and a fixed record size. The circular buffer implemented the get/put functions to make sure that you roll over the end of the 'array' on the SD card. You always write to the 'put' and read from the 'get'. Put must also implement some logic for the roll over to overwrite the oldest value, but it is straightforward stuff.

Define a C(++) structure that contains all the data you need for a 'point'. Then each record will be a set offset from the start of the file (item j is j*record_size from the start of the SD card file). Seek to that position in the file to read/write the structure (as binary data).

Alternatively, if you have enough RAM why not keep the last 300 values in an array in memory?

After some thinking I have settled on reading new data every 15 seconds over 5 minutes rather than every second, which cuts down my data pool from 300 to 20. Because now I only have 20 reads over 12 sensor, I can make use of a set of arrays instead of storing everything on the SD (the array will most likely be written to an I2C EEPROM).

Here is what I came up with for rolling the new readings in and the old ones out.

byte swapArray[20];
byte tempArray[20];

  for(int index = 0; index < 19; index++)
  {
    swapArray[index] = tempArray[index + 1];
  }

  memcpy (tempArray, swapArray, 20);
  tempArray[19] = 30; //The "new" data

I was fighting a bit of boredom and and wanted to practice for() loops so… the proof is in the puddling.

/*
A little program to roll new elements into
an existing array and to delete the oldest ones

I wrote this so that I can use it to create
a scrolling graph on an LCD screen keeping the
"x" scale of the screen fixed.
*/

byte swapArray[20];
byte Array[20];

void setup()
{
  Serial.begin(9600);
  Serial.println("Start!!! Delay 2 seconds");

  delay(2000);

  Serial.print("Populating tempArray ");
  for(int index = 0; index < 20; index++)
  {
    tempArray[index] = index;
  }

  Serial.print("swapArray = ");
  for(int index = 0; index < 20; index++)
  {
    Serial.print(swapArray[index]); 
    Serial.print(",");
    delay(250);
  }

  Serial.print("Array = ");
  for(int index = 0; index < 20; index++)
  {
    Serial.print(tempArray[index]); 
    Serial.print(",");
    delay(250);
  }

  for(int index = 0; index < 19; index++)
  {
    swapArray[index] = tempArray[index + 1];
  }

  Serial.print("New swapArray = ");
  for(int index = 0; index < 20; index++)
  {
    Serial.print(swapArray[index]); 
    Serial.print(",");
    delay(250);
  }

  memcpy (tempArray, swapArray, 20);

  Serial.print("New Array = ");
  for(int index = 0; index < 20; index++)
  {
    Serial.print(tempArray[index]); 
    Serial.print(",");
    delay(250);
  }

  tempArray[19] = 30; //Writing in the "new" data

  Serial.print("Final Array = ");

  for(int index = 0; index < 20; index++)
  {
    Serial.print(tempArray[index]); 
    Serial.print(",");
    delay(250);
  }
  Serial.println("Done!");
}

void loop()
{
}