Removing data from a file on a SD Card

I'm just beyond the concept phase for a data logging and control project that once started will have minimal to no downtime in order to remove the SD card to retrieve the data. Reading the data log from the card will be done via wireless USB module. The system will also be logging to the card at the same time as data is being read. The plan is to read several lines into a transmit buffer then close the file and allow for the file to be appended. When the transmit buffer is empty, I need to remove the first number of already transmitted lines from the file. This is where I'm stuck. How do I remove data from a file on an SD card? Whether it is at the beginning (like this situation), in the middle or at the end?

(Before you ask, I don't have a working sketch yet. I'm still gathering the pieces of code necessary to make it work, let alone compile)

I could answer your question, but I will ask you to explain your plan for when the Arduino has a power failure or when the SD card file is full.

Paul

When the card is full, the oldest file will be deleted.

As for power failure. If the power goes out then everything goes out and there isn't anything to data log. If the power goes out while sending data, then two sets of the same data will be sent but at least it won't be lost. The final aspect will be a power failure while trying to remove the old data, but since I haven't figured out how to do that I cannot perform the FMEA on it.

adwsystems:
When the card is full, the oldest file will be deleted.

As for power failure. If the power goes out then everything goes out and there isn't anything to data log. If the power goes out while sending data, then two sets of the same data will be sent but at least it won't be lost. The final aspect will be a power failure while trying to remove the old data, but since I haven't figured out how to do that I cannot perform the FMEA on it.

Then your plan is to make each write to the SD card a separate file? You need to close the current file after each data write and open a new one?

Paul

Paul_KD7HB:
Then your plan is to make each write to the SD card a separate file? You need to close the current file after each data write and open a new one?

Paul

Not sure you extrapolated one entry per file. Open file, write data line, close file. Short and quick. To avoid have one HUGE file, I will check the file size when I open it, if it exceeds the limit, I will write the line, close the file, increment the file name 'index', open the new file, write the header. The limit will be in MB, I just haven't decided on the exact number yet. Somewhere between 1 and 100.

Oops. Correction to post 1. I don't need to erase lines 1-5, but 2-6. Line 1 is a header line.

The only way I know of to remove lines from the start of a file is to make a copy of the file line by line, ignoring the lines you want to drop. Then delete the original and rename the new as the old.

I'm not really clear why you need to though. If you're trying to ensure that you don't retransmit data, you can overwrite it with something that tells your transmit routine to ignore it. Then, at the end of your reporting period, once you're sure all the data has been sent, delete the file.

wildbill:
The only way I know of to remove lines from the start of a file is to make a copy of the file line by line, ignoring the lines you want to drop. Then delete the original and rename the new as the old.

That's what I was afraid off.

wildbill:
If you're trying to ensure that you don't retransmit data, you can overwrite it with something that tells your transmit routine to ignore it. Then, at the end of your reporting period, once you're sure all the data has been sent, delete the file.

Quite possibly an option. How would I go about overwriting lines 2 through 6? I haven't seen an example of that yet, and haven't digested all of the functions and usage available in the SDFat library.

Are your lines fixed length? If so, seek() would be helpful. If not, you'll need to read line by line and overwrite as you go.

The first line will not be the same length as the data entries. I'm still developing the write_datalog() function. So far the data lines are the same length, but let's think the more difficult route they aren't the same length. It is fine to read line by line. One line is one time stamped entry, so that works nicely.

Step 1: Read line 1 (header)
Step 2a: Read line 2 (the first line to be discarded)
Step 2b: Overwrite line 2
Step 3a-b: Repeat steps 2a-b for lines 3, 4, 5, and 6
Step 4: Read lines next 5 lines to be transmitted

How is step 2b to be accomplished?

I have figured out how to delete a file, write to the end of a file, even copy a file line by line. I'm not yet seeing how to write to the middle of an existing file (opposed to read/write to copy to another file).

If your records are all variable length, how would find the "middle of the file"? OR even identify it?

Paul

Based on post #7 from wildbill, it appears he has an idea. I'm not sure what it is. He mentions "reading line by line" if the records are variable length. As the data records are line by line, his train of thought looks promising. I will always know which and how many records to delete (the 5 lines of the file, #2 through #6).

It’s messy. You need to use the position and seek functions. Read the file character by character noting when you have reached the end of a line. Record where you are in the file using position. If this is a line you wish to transmit, read it into a buffer and then use seek to return to the start of the line. Write some characters there to mark the line as no longer needed. Read to the end of line again and continue. You could use position at the end of the line too and seek to get back there.

It’s ugly but doable. Since you don’t wish to keep the data on the SD card though, I’d be more inclined to close the log file when you want to transmit and open a new one to log in, deleting the transmission file when it’s been received.

Yes, starting a new file if transmitting the current logfile is a viable thing to do. The file name convention selected would allow for that.

In reading the examples files installed with the SDFat library, I noted four different ways to read data from the file on the SD card. (and I'm sure there are more that aren't covered in the examples and I don't know about)

  1. SdFile fgets
  2. ifstream getline
  3. SdFile read
  4. SdFile get

Option 2 has my attention as it reads only one line and the whole line (up to the char size). I have rarely used streams and don't recall using ifstream before. According to ifstream - C++ Reference there are also a tellg and seekg functions. Would these functions be viable for producing the method as described in post 11?

Since I have already read the lines to be transmitted, I can record the start position and length of each line as I read them. I can skip the step where I have to jump back to start the overwrite. The sequence would be to read line, does position equal any of the previous read start positions if yes, then erase data. If not, then read data until read data buffer is full.

Question on position value: What size (type) of variable is returned from the ifstream tellg and SdFile curPosition? I would think this is a character position within the file, and therefore the file cannot have more characters than the position value can hold.

If I "erase" the data already transmitted in this method, it appears the erase will just be overwriting the old data with spaces (or other character)? Therefore the transmitted file and the original file would be the same length?

They will indeed be the same length, which may defeat what you're trying to do. I assume that the SD card is intended to hold the data until you get close enough to the device with a wifi equipped reader.

If so, can you simply equip it with a big enough card that any considerations of saving space are irrelevant?

wildbill:
They will indeed be the same length, which may defeat what you're trying to do.

No. Not at all. Just working to understand the implementation and implications.

Do you have any information on using ifstream or on the value size for the position?

The Arduino SD library returns unsigned long for position. I'd expect SDFat to provide that or more.

wildbill:
The Arduino SD library returns unsigned long for position. I’d expect SDFat to provide that or more.

So limited to a 4GB file. Not that I would want nor plan to have a single file that large.

Yes I’m using SDFat. The getline is part of ifstream as is tellg and seekg. Is the code using SDFat?

/*
 * Example of getline from section 27.7.1.3 of the C++ standard
 * Demonstrates the behavior of getline for various exceptions.
 * See http://www.cplusplus.com/reference/iostream/istream/getline/
 *
 * Note: This example is meant to demonstrate subtleties the standard and
 * may not the best way to read a file.
 */
#include <SPI.h>
#include "SdFat.h"

// SD chip select pin
const uint8_t chipSelect = SS;

// file system object
SdFat sd;

// create a serial stream
ArduinoOutStream cout(Serial);
//------------------------------------------------------------------------------
void makeTestFile() {
  ofstream sdout("getline.txt");
  // use flash for text to save RAM
  sdout << F(
          "short line\n"
          "\n"
          "17 character line\n"
          "too long for buffer\n"
          "line with no nl");

  sdout.close();
}
//------------------------------------------------------------------------------
void testGetline() {
  const int line_buffer_size = 18;
  char buffer[line_buffer_size];
  ifstream sdin("getline.txt");
  int line_number = 0;

  while (sdin.getline(buffer, line_buffer_size, '\n') || sdin.gcount()) {
    int count = sdin.gcount();
    if (sdin.fail()) {
      cout << "Partial long line";
      sdin.clear(sdin.rdstate() & ~ios_base::failbit);
    } else if (sdin.eof()) {
      cout << "Partial final line";  // sdin.fail() is false
    } else {
      count--;  // Don’t include newline in count
      cout << "Line " << ++line_number;
    }
    cout << " (" << count << " chars): " << buffer << endl;
  }
}
//------------------------------------------------------------------------------
void setup(void) {
  Serial.begin(9600);
  
  // Wait for USB Serial 
  while (!Serial) {
    SysCall::yield();
  }

  // F stores strings in flash to save RAM
  cout << F("Type any character to start\n");
  while (!Serial.available()) {
    SysCall::yield();
  }

  // Initialize at the highest speed supported by the board that is
  // not over 50 MHz. Try a lower speed if SPI errors occur.
  if (!sd.begin(chipSelect, SD_SCK_MHZ(50))) {
    sd.initErrorHalt();
  }

  // make the test file
  makeTestFile();

  // run the example
  testGetline();
  cout << "\nDone!\n";
}
//------------------------------------------------------------------------------
void loop(void) {}