How can I find and read the last lines of a large .txt file on an SD card?

Hello,
First some project info...
I'm building a small pressure data logger for my agricultural sprayer. It contains a Nano, pressure sensor, RTC clock, SD card, 16x2 LCD and one button.
Every second the pressure sensor is read, displayed on the LCD and stored in a 10 places array.
Every 10th second the average of the 10 last measurements are stored on the SD card including date and time. for that a made a 16 character String in the format "ddmm_hhmm:ss_p,p" . p,p means pressure from 0,0 between 9,9 bar. if the day or time is less than 10 an extra "0" is added, so I have always a 16 letter string. 360 lines for each hour of sprayer activity makes a (long) data list on my "datalog.txt" file. But a 32Gb card can easily store a year of work.
I use the SD.h library for access of the SD card
the code (for now) is in the attachment.

And then my question:

Now I want (with one push on the button) the last 360 lines (the last hour work) data one by one (for one second) back on my 16x2 LCD display. So I have to find the end of the file, and then go one line up at a time.

Do I need a different library with more functions?? Do I have to write the file upside down: the last log as first line instead of the last line??

I have no idea how to start with this, and can't find a proper example on arduino.cc, youtube or google. This question is already solved somewhere by someone, I'm sure, but where can I find it?

drukregistratie5.ino (9.14 KB)

I have not use an SD card. But, if your record lengths are ALL identical, you can use a random access method by seeking a record number and reading that record.

Research it.

Paul

bertschermers:
Do I need a different library with more functions??

I don't think there would be one that does exactly what you want.

bertschermers:
Do I have to write the file upside down: the last log as first line instead of the last line??

That's practically impossible. You can either overwrite or append, but not insert. The library is not a text editor, in fact it treats all files as binary (as an array of bytes).

So you have two options: the slow one and the fast one.

The slow one consists of seeking the end of the file and then reading backwards untill you find the enough amount of line separators.
The fast one consists of making fixed-length records, so the random access would be as simple as in an array (as suggested by Paul_KD7HB).

Hello,

I have found the answer for the problem: using SD.size() and SD.seek() is the solution.
but this will only work with a fixed String length

I made a testdatabase of 2500 lines and the following code:

myFile = SD.open("TESTDATA.txt", FILE_READ);    //file open, in read modus
  //  Serial.println(myFile.size());                       // just to see the filesize... 
 aantal = myFile.size()-1800;                            //1 line is 18 char, so 100 lines equals minus 1800
  myFile.seek(aantal);                                     // the cursor goes to 100 regels before the end-of-file

  for (int i=0;i<100;i++){                                 // now we go read the next 100 lines
 
 buffer=myFile.readStringUntil('\n');                 // String read in "buffer"
  lcd.setCursor(0,1);                    //
  lcd.print(buffer);                                        // print string on LCD 
  delay(1000);                                             // wait  second: time to read the LCD
  }

...and it shows the database lines starting with 2401 to 2500

Thanks a lot for your replies!!

2 Likes

bertschermers:
but this will only work with a fixed String length.

That's what Paul_KD7HB and I were talking about; now you know why.

bertschermers:
I made a testdatabase of 2500 lines and the following code:

myFile = SD.open("TESTDATA.txt", FILE_READ);    //file open, in read modus

//  Serial.println(myFile.size());                       // just to see the filesize...
aantal = myFile.size()-1800;                            //1 line is 18 char, so 100 lines equals minus 1800
 myFile.seek(aantal);                                     // the cursor goes to 100 regels before the end-of-file

for (int i=0;i<100;i++){                                 // now we go read the next 100 lines

buffer=myFile.readStringUntil('\n');                 // String read in "buffer"
 lcd.setCursor(0,1);                    //
 lcd.print(buffer);                                        // print string on LCD
 delay(1000);                                             // wait  second: time to read the LCD
 }

Now I'm wondering: do you even fill the slack with something?
I mean: your records have a length of 18 bytes, but what if the resulting string is shorter? Do you fill the remaining space with zeroes or whitespaces until you can finally place the new line characters?

hello lucario448, thanks for the reply,

I made a string (record) for one line on a 16x2 LCD with date time and pressure (with comma and one decimal):

ddmm_hhmm:ss_p,p

the gaps in date and time when the numbers are <10 were filled with a "0" (zero), so the string is always exact 16 caracters

I made a testfile with 2500lines of 16 characters and count the characters with SD.size()
when I divided the outcome by 2500, it says 18!!
One extra char should be the carriage return, the last one I really do'nt know.

But when I want 100 records back, I subtract 1800, en the lines are exactly 16 long on the display.

This part works as I want for my project now, so I go on working my datalogger, I'm not going further at this moment with what-if questions. I'll figure that out maybe on a following project when I need it in my code.

bertschermers:
I made a testfile with 2500lines of 16 characters and count the characters with SD.size()
when I divided the outcome by 2500, it says 18!!
One extra char should be the carriage return, the last one I really do'nt know.

Every line ends with the two-character sequence "\r\n" (carriage return and line feed) when you use println().

that explains why every 16 number record counts 18 characters.
I think that's a lack of my basic programming knowledge...
Thanks!!

a 16 character String in the format "ddmm_hhmm:ss_p,p"

how I cut 8 characters per file: I make the file name "YEARMMDD.CSV" There is no point saving the date in every file when the file name can serve as the date, and the file name is automatically generated or opened when you write.

here's the windup:

// SD & Datalogger
const int chipSelect = 53;          // SPI CS for SD on a Mega
File dataFile;                      // variable for SD
char filename[]  = "00000000.CSV";  // filename array for datalogger

...and the pitch:

void getFilename( time_t LocalTime )
     {          
      sprintf(filename, "%04d%02d%02d", year(LocalTime), month(LocalTime), day(LocalTime));
     }

you can substitute rtc.get.year(); or whatever works with your RTC library for LocalTime

the function invoked:

void logEvent(int sw)
{
 time_t eventTime = LocalTime();
      {
       getFilename( eventTime );
       File dataFile = SD.open(filename, FILE_WRITE);
       if (dataFile)                                                   // if the file is available, write to it:

( you know the rest )

that jibberish in blue in the line

sprintf(filename, "%04d%02d%02d", year(LocalTime), month(LocalTime), day(LocalTime));

formats the display:

%: pad it to a fixed length
0: use leading 0s to pad it out
4: 4 digits for the year, 2 each for month and day
d: display in digital format, vs hex, octal...

study "sprintf". way better than endless repetitions of "if value is <10 print"0"...

I have found the answer for the problem: using SD.size() and SD.seek() is the solution.

did you have to use SdFat.h to make that work? this is why people ask to see all the code