Speed issue reading a particular line in large tab-delimited files on SD card

Other Hardware Storage
1

Recently, I wrote code for reading a particular entry from the log-file stored on an SD card. The example file was exported as tab-delimited text file from Microsoft Excel.

The code works as follows:

  1. The indexing function is called with line number as argument.
  2. It reads the file one character at a time.
  3. Counts the number of line termination characters (ASCII INT value = 10) and compares against the argument.
  4. When the line of interest is reached, the whole line is copied into a global character string.
  5. The global character string is then parsed by another function to extract numbers and text.

The code works well for files containing upto 100 or so lines. However, it becomes painfully slow when it comes to data files containing thousands of lines, as each time, the code reads all characters from the beginning till the line of interest.

Thanks in advance.

2

Can you force the file to have a fixed number of characters per line?

3

Yes - that can always be done by filling the rows with non-printable characters, such as white spaces. Your idea seems to be making some sense - if only there exists a way to jump to some specific location in the file.

Besides, I am also concerned about filling Arduino's memory while reading the file, even if we are reading just one character at a time, I am afraid, more of it would be getting loaded into the working memory. So far, the program has successfully handled 175kB file on Uno. Will have to perform further experimentation before something can be concluded.

4

Look at the seek command.

5

Thanks for the suggestion. I have implemented and it has really improved the speed. Here is the complete function for benefit of others:

void read_SD_file_position_2_string_SD_file_data(int line_num /*Works with one header row*/) /*Opens global string_SD_file_name[13] and saves the required 
line to global string string_SD_file_data[] which is then parsed by another function.*/
{
  strcpy(string_SD_file_data,"");
  int character_location=0;
  SD_file = SD.open(string_SD_file_name);
  if (SD_file)
  {
    // read from specific location in the file
    SD_file.seek(24+(line_num-1)*58);
    Serial.print('\n'); //Serial.print(character);
    Serial.print("Position: ");
    Serial.print(SD_file.position()); //Serial.print(character);
    while ((SD_character = SD_file.read()) != 10) //Till next blank in tab delimited file; 9=TAB; 10=Next line
    {
      string_SD_file_data[character_location]=SD_character; //This syntax will work only for strings having pre-defined length
      character_location+=1;
    }
    SD_file.close();
    string_SD_file_data_parser(); //Call string parser
  }
  else
  {
    Serial.println("Err opening ");
    Serial.println(string_SD_file_name);
  }
}