Fast access inside a file with SDFat -- best method?

Other Hardware Storage
1

Dear all,

I have a large (several thousand lines) CSV log file to handle from an SD card, but I just need to check the content of a field (log time) and read a single line (most often it will be the last line). This field contains a time tag (UNIX time) so I know the lines are ordered.

It takes a lot of time if we read and parse the file line by line. I know that there is an extremely fast method to access sorted data (Binary search algorithm - Wikipedia) and I would like to implement it.

I would like to get your advice: what would be the fastest method to access the line I'm looking for?
The total number of lines is unknown, but this is what is known:

  • file size
  • approximation of average number of characters per line
  • data structure (CSV)

I'm using SDFat lib, which contains many methods to access data. Which one is the best? :~
Thanks for your help! :grin:

2

what would be the fastest method to access the line I'm looking for?

If you know that the record of interest is near the end of the file, go to the end of the file (file.seek()), move back some number of characters - the average record length times some small number of records (again using seek()), and then read (forward) until you find the start of the next record. Read as usual, but having to read far less of the file.

3

Indeed seek() seems to be the instruction I was looking for! :grin:

Thank you very much !!! XD