Read SD file into an Array

Using Arduino Storage
1

Purpose: To relate some (hard for me) lessons learned in getting a file from an SD card into an array.
Background: Making an in-car computer for high speed rally events. File on SD card contains target speed and some waypoints (mileages, lat/long) for each specific race course.
Hardware: Arduino MKR Zero.
Key Reference: A Simple Function for Reading CSV Text Files. - #3 by fat16lib A beautifully crafted program from fat16lib. I read many posts before stumbling upon this outstanding approach from fat16lib that works! Yes, I modified it a bit, but what a great resource!
Lessons Learned:

  1. File names limited to 8 characters!
  2. Add code to wait for Serial port to open (see code sample appended)
  3. Array rows and columns are numbered starting at 0 (not 1)
  4. Wait for uploading to complete before trying to open Serial Monitor
  5. When your MKR Zero never ends the uploading, hit the reset button twice in quick succession (starts the bootloader, magically fixes things)
  6. Integers in MKR Zero are NOT limited to +-32,768 like in the Uno
  7. BTW, the lovely MKR GPS shield has such a hard time getting a GPS fix that I abandoned it and used an Adafruit Ultimate GPS Breakout instead. You can put an external antenna on it! Imagine being at a race start line and not getting a GPS reading.
  8. The CS PIN for the SD on the MKR Zero is NOT 4 and NOT 10 (I think it's 28) - but just specify it as SDCARD_SS_PIN. For example,
    if (!SD.begin (SDCARD_SS_PIN))
    {
    etc.
  9. You have to close a file before opening a new one.

The code is appended at the end. This is the output on the Serial Monitor:
130 0 38 -115 28 38 -114 60 37 -115 61 37 -115 0
0 0 433614 -13378 61 41172 -967558 811 634958 -21725 380 627386 -271253 0
38.433614
130 0 37 -115 32 38 -114 60 38 -115 61 38 -115 0
0 0 627386 -221253 182 25078 -963764 622 424125 -20431 190 43125 -1515 0
37.627386
startLat should be 38.433614 and is actually: 38.433614
startLng should be -115.13378 and is actually: -115.013378
Done
MarkIIISD2ArraysTest.ino (7.1 KB)

// Example to read a two dimensional array.
// Example to read a two dimensional array.
//
#include <SPI.h>
#include <SD.h>
#define CS_PIN SDCARD_SS_PIN

// 2 X 14 array
#define ROW_DIM 2
#define COL_DIM 14

File file;
double startLat = 0.0;
int startLatD = 0;
double startLatM = 0.0;
double startLng = 0.0;
int startLngD = 0;
double startLngM = 0.0;

/*
   Note: Integers in MKR Zero have a range of -2,147,483,648 to + 2,147,483,647
   Read a file one field at a time.
   NOTE: File names are LIMITED to 8 CHARACTERS!!

   file - File to read.

   str - Character array for the field.

   size - Size of str array.

   delim - String containing field delimiters.

   return - length of field including terminating delimiter.

   Note, the last character of str will not be a delimiter if
   a read error occurs, the field is too long, or the file
   does not end with a delimiter.  Consider this an error
   if not at end-of-file.

*/
size_t readField(File* file, char* str, size_t size, char* delim) {
  char ch;
  size_t n = 0;
  while ((n + 1) < size && file->read(&ch, 1) == 1) {
    // Delete CR.
    if (ch == '\r') {
      continue;
    }
    str[n++] = ch;
    if (strchr(delim, ch)) {
      break;
    }
  }
  str[n] = '\0';
  return n;
}
//------------------------------------------------------------------------------
#define errorHalt(msg) {Serial.println(F(msg)); while(1);}
//------------------------------------------------------------------------------
void setup() {

  Serial.begin(115200); // Note that I use 115200 baud although 9600 is more common
  // Open serial communications and wait for port to open: ADD THIS WAIT OR NATIVE USB PORT WON'T OPEN
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // Initialize the SD.
  if (!SD.begin(SDCARD_SS_PIN)) // This is where you specify the CS Pin for MKR Zero
    // Yes, I know, I did a #define up above, but I wanted to make it clear and specific here
  {
    errorHalt("begin failed");
  }
  // Open the file.

  file = SD.open("N130L1.txt", FILE_WRITE); // File names limited to 8 characters!
  if (!file) {
    errorHalt("first open failed");  // You'll see this message if your file name > 8 chars
  }

  // Rewind file so test data is not appended.
  file.seek(0);  // Who knew there was a "seek" function?  This is useful.

  // Write test data.  Note that the first row has the most significant parts of the data, and the
  // second row the least significant parts.  For example, target speed is 130.0 mph.  Start line is at
  // mileage 0.000.  Lat/Lng is 38.433614, -115.433614.
  file.print(F(
               "130,0,38,-115,28,38,-114,60,37,-115,61,37,-115,0\r\n"
               "0,0,433614,-13378,61,41172,-967558,811,634958,-21725,380,627386,-271253,0\r\n"
             ));
  /*
     This is fat16lib's original example, much more general than my specific file
    "11,12,13,14\r\n"
    "21,22,23,24\r\n"
    "31,32,33,34\r\n"
    "41,42,43,44\r\n"
    "51,52,53,54"     // Allow missing endl at eof.

    ));
  */
  // Rewind the file for read.
  file.seek(0);

  // Array for data.
  long arrayL1[ROW_DIM][COL_DIM];
  int i = 0;     // First array index.
  int j = 0;     // Second array index
  size_t n;      // Length of returned field with delimiter.
  char str[20];  // Must hold longest field with delimiter and zero byte.
  char *ptr;     // Test for valid field.

  // Read the file and store the data.

  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      n = readField(&file, str, sizeof(str), ",\n");
      if (n == 0) {
        errorHalt("Too few lines");
      }
      arrayL1[i][j] = strtol(str, &ptr, 10);
      if (ptr == str) {
        errorHalt("bad number");
      }
      if (j < (COL_DIM - 1) && str[n - 1] != ',') {
        errorHalt("line with too few fields");
      }
    }
    // Allow missing endl at eof.
    if (str[n - 1] != '\n' && file.available()) {
      errorHalt("missing endl");
    }
  }

  // Print the array.
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      if (j) {
        Serial.print(' ');
      }
      Serial.print(arrayL1[i][j]);
    }
    Serial.println();
  }
  //Some samples: extract from the arrays and print (you can also put values into internal variables)
  Serial.print(arrayL1[0][2]); // Remember that rows and columns begin at 0
  Serial.print(".");
  Serial.println(arrayL1[1][2]);
  file.close(); // You have to close the file before trying to open a new one.

  // Open the second file.  In a race, this is the inbound leg of the event.

  file = SD.open("N130L2.txt", FILE_WRITE); // File names limited to 8 characters!
  if (!file) {
    errorHalt("second open failed");  // You'll see this message if your file name > 8 chars
  }

  // Rewind file so test data is not appended.
  file.seek(0);  // Who knew there was a "seek" function?  This is useful.

  // Write test data for the inbound leg of the race.
  file.print(F(

               "130, 0, 37, -115, 32, 38, -114, 60, 38, -115, 61, 37, -115, 0\r\n"
               "0, 0, 627386, -221253, 182, 25078, -963764, 622, 424125, -20431, 190, 43125, -1515, 0\r\n"
             ));

  //Now read the second leg file into another array

  // Rewind the file for read.
  file.seek(0);

  // Array for data.
  long arrayL2[ROW_DIM][COL_DIM];
  i = 0;     // First array index.
  j = 0;     // Second array index

  // Read the file and store the data.

  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      n = readField(&file, str, sizeof(str), ",\n");
      if (n == 0) {
        errorHalt("Too few lines");
      }
      arrayL2[i][j] = strtol(str, &ptr, 10);
      if (ptr == str) {
        errorHalt("bad number");
      }
      if (j < (COL_DIM - 1) && str[n - 1] != ',') {
        errorHalt("line with too few fields");
      }
    }
    // Allow missing endl at eof.
    if (str[n - 1] != '\n' && file.available()) {
      errorHalt("missing endl");
    }
  }

  // Print the array.
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      if (j) {
        Serial.print(' ');
      }
      Serial.print(arrayL2[i][j]);
    }
    Serial.println();
  }
  Serial.print(arrayL2[0][2]); // Remember that rows and columns begin at 0
  Serial.print(".");
  Serial.println(arrayL2[1][2]);

  // Nice thing is you can manipulate the data from the SD file in your program:
  startLatD = (arrayL1[0][2]);
  startLatM = (arrayL1[1][2]);
  startLat = (startLatD + (startLatM / 1000000));

  startLngD = (arrayL1[0][3]);
  startLngM = (arrayL1[1][3]);
  startLng = (startLngD + (startLngM / 1000000)); //NOTE: enter LS longitude as negative nos.

  Serial.print("startLat should be 38.433614 and is actually: ");
  Serial.println(startLat, 6);
  Serial.print("startLng should be -115.13378 and is actually: ");
  Serial.println(startLng, 6);

  Serial.println("Done");
  file.close();

}

//------------------------------------------------------------------------------
void loop() {
}
2

The easier you make it to read and copy your code the more likely it is that you will get help

Please follow the advice given in the link below when posting code , use code tags and post the code here

3

Thanks for sharing your experiences.

Actually, they need to adhere to the 8dot3 notation; max 8. for the base filename, max. 3 for the extension.

That depends on the aplication; what are you going to do if you e.g. have a display on your board that will display the info and the Serial Monitor is just an additional 'luxury'?

One learns the hard way :wink:

Yes, it's a 32-bit processor :slight_smile: For compatible code between 8-bit and 32-bit processors, use e.g. int16_t or uint32_t and so on; those will be the same size between the different architectures.

I include your code in this post (hope you don't mind) so it's straight away available to people for viewing; I've only fixed the indentations. Nice to see somebody use size_t :wink:

// Example to read a two dimensional array.
//
#include <SPI.h>
#include <SD.h>
#define CS_PIN SDCARD_SS_PIN

// 2 X 14 array
#define ROW_DIM 2
#define COL_DIM 14

File file;
double startLat = 0.0;
int startLatD = 0;
double startLatM = 0.0;
double startLng = 0.0;
int startLngD = 0;
double startLngM = 0.0;

/*
   Note: Integers in MKR Zero have a range of -2,147,483,648 to + 2,147,483,647
   Read a file one field at a time.
   NOTE: File names are LIMITED to 8 CHARACTERS!!

   file - File to read.

   str - Character array for the field.

   size - Size of str array.

   delim - String containing field delimiters.

   return - length of field including terminating delimiter.

   Note, the last character of str will not be a delimiter if
   a read error occurs, the field is too long, or the file
   does not end with a delimiter.  Consider this an error
   if not at end-of-file.

*/
size_t readField(File* file, char* str, size_t size, char* delim) {
  char ch;
  size_t n = 0;
  while ((n + 1) < size && file->read(&ch, 1) == 1) {
    // Delete CR.
    if (ch == '\r') {
      continue;
    }
    str[n++] = ch;
    if (strchr(delim, ch)) {
      break;
    }
  }
  str[n] = '\0';
  return n;
}
//------------------------------------------------------------------------------
#define errorHalt(msg) {Serial.println(F(msg)); while(1);}
//------------------------------------------------------------------------------
void setup() {

  Serial.begin(115200); // Note that I use 115200 baud although 9600 is more common
  // Open serial communications and wait for port to open: ADD THIS WAIT OR NATIVE USB PORT WON'T OPEN
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // Initialize the SD.
  if (!SD.begin(SDCARD_SS_PIN)) // This is where you specify the CS Pin for MKR Zero
    // Yes, I know, I did a #define up above, but I wanted to make it clear and specific here
  {
    errorHalt("begin failed");
  }
  // Open the file.

  file = SD.open("N130L1.txt", FILE_WRITE); // File names limited to 8 characters!
  if (!file) {
    errorHalt("first open failed");  // You'll see this message if your file name > 8 chars
  }

  // Rewind file so test data is not appended.
  file.seek(0);  // Who knew there was a "seek" function?  This is useful.

  // Write test data.  Note that the first row has the most significant parts of the data, and the
  // second row the least significant parts.  For example, target speed is 130.0 mph.  Start line is at
  // mileage 0.000.  Lat/Lng is 38.433614, -115.433614.
  file.print(F(
               "130,0,38,-115,28,38,-114,60,37,-115,61,37,-115,0\r\n"
               "0,0,433614,-13378,61,41172,-967558,811,634958,-21725,380,627386,-271253,0\r\n"
             ));
  /*
     This is fat16lib's original example, much more general than my specific file
    "11,12,13,14\r\n"
    "21,22,23,24\r\n"
    "31,32,33,34\r\n"
    "41,42,43,44\r\n"
    "51,52,53,54"     // Allow missing endl at eof.

    ));
  */
  // Rewind the file for read.
  file.seek(0);

  // Array for data.
  long arrayL1[ROW_DIM][COL_DIM];
  int i = 0;     // First array index.
  int j = 0;     // Second array index
  size_t n;      // Length of returned field with delimiter.
  char str[20];  // Must hold longest field with delimiter and zero byte.
  char *ptr;     // Test for valid field.

  // Read the file and store the data.

  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      n = readField(&file, str, sizeof(str), ",\n");
      if (n == 0) {
        errorHalt("Too few lines");
      }
      arrayL1[i][j] = strtol(str, &ptr, 10);
      if (ptr == str) {
        errorHalt("bad number");
      }
      if (j < (COL_DIM - 1) && str[n - 1] != ',') {
        errorHalt("line with too few fields");
      }
    }
    // Allow missing endl at eof.
    if (str[n - 1] != '\n' && file.available()) {
      errorHalt("missing endl");
    }
  }

  // Print the array.
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      if (j) {
        Serial.print(' ');
      }
      Serial.print(arrayL1[i][j]);
    }
    Serial.println();
  }
  //Some samples: extract from the arrays and print (you can also put values into internal variables)
  Serial.print(arrayL1[0][2]); // Remember that rows and columns begin at 0
  Serial.print(".");
  Serial.println(arrayL1[1][2]);
  file.close(); // You have to close the file before trying to open a new one.

  // Open the second file.  In a race, this is the inbound leg of the event.

  file = SD.open("N130L2.txt", FILE_WRITE); // File names limited to 8 characters!
  if (!file) {
    errorHalt("second open failed");  // You'll see this message if your file name > 8 chars
  }

  // Rewind file so test data is not appended.
  file.seek(0);  // Who knew there was a "seek" function?  This is useful.

  // Write test data for the inbound leg of the race.
  file.print(F(

               "130, 0, 37, -115, 32, 38, -114, 60, 38, -115, 61, 37, -115, 0\r\n"
               "0, 0, 627386, -221253, 182, 25078, -963764, 622, 424125, -20431, 190, 43125, -1515, 0\r\n"
             ));

  //Now read the second leg file into another array

  // Rewind the file for read.
  file.seek(0);

  // Array for data.
  long arrayL2[ROW_DIM][COL_DIM];
  i = 0;     // First array index.
  j = 0;     // Second array index

  // Read the file and store the data.

  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      n = readField(&file, str, sizeof(str), ",\n");
      if (n == 0) {
        errorHalt("Too few lines");
      }
      arrayL2[i][j] = strtol(str, &ptr, 10);
      if (ptr == str) {
        errorHalt("bad number");
      }
      if (j < (COL_DIM - 1) && str[n - 1] != ',') {
        errorHalt("line with too few fields");
      }
    }
    // Allow missing endl at eof.
    if (str[n - 1] != '\n' && file.available()) {
      errorHalt("missing endl");
    }
  }

  // Print the array.
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      if (j) {
        Serial.print(' ');
      }
      Serial.print(arrayL2[i][j]);
    }
    Serial.println();
  }
  Serial.print(arrayL2[0][2]); // Remember that rows and columns begin at 0
  Serial.print(".");
  Serial.println(arrayL2[1][2]);

  // Nice thing is you can manipulate the data from the SD file in your program:
  startLatD = (arrayL1[0][2]);
  startLatM = (arrayL1[1][2]);
  startLat = (startLatD + (startLatM / 1000000));

  startLngD = (arrayL1[0][3]);
  startLngM = (arrayL1[1][3]);
  startLng = (startLngD + (startLngM / 1000000)); //NOTE: enter LS longitude as negative nos.

  Serial.print("startLat should be 38.433614 and is actually: ");
  Serial.println(startLat, 6);
  Serial.print("startLng should be -115.13378 and is actually: ");
  Serial.println(startLng, 6);

  Serial.println("Done");
  file.close();

}

//------------------------------------------------------------------------------
void loop() {
}
4

UKHeliBob, I totally agree with making the code easily accessible. The problem I had when posting was that the new layout of the Forum didn't seem to have the "embed code" option any longer (only an "embed date & time"), so I didn't know exactly how to do it properly. And, sterretje, thanks for posting the code and cleaning it up and clarifying those technical points. In answer to your good question about the Serial Monitor, I was only using it for testing; but since it was nicely used in fat16lib's code, I thought it would be fair to show people how to avoid the frustration of never seeing anything appear in the Serial Monitor if they are using the native USB port but not waiting for it to open. Anyway, thanks to all for the quick and expert feedback!

5

The icon for code tags is exactly the same as it always was </> although its description is not as precise.

However, no doubt you will have read How to get the best out of this forum which describes a number of ways to post code properly

6

Thank you, UKHeliBob. I saw that symbol and hovering over it showed "preformatted text" rather than "embed code." So you can see that I'm easily confused. In any case, I have taken your expert input and edited my original post to make the code more accessible I hope. Cheers.

7

Thanks for adding the code tags

There has been some debate about changing the text of the tooltip to make its purpose here more obvious but the discussion seems to have dried up. I will try to revive it

8

It did not quite work out :wink:

If you want to fix it,

  1. Replace the single backticks that you used by triple backticks
  2. The triple backticks must be on their own line

You can quote my post and check in the editor box how I have done it. Next you can discard the quoted post.

9

sterretje, you are very kind. I couldn't quite follow the backtick explanation because I'm somewhat dense, so I copied your nicely edited code and stuck that in. I hope it works. In any case, I greatly appreciate your gentle nudging to steer me in the right direction to help others as they struggle with reading an SD file into an array. I consider you a great asset to the Forum!

10

It didn't fix it by the looks of it :wink:

This is a single backtick: `
This is a triple backtick: ```

Replace the single ones (the one I showed in the quited text and the one at the end of the code) by the triple ones; they also need to be placed on a new line and the code needs to start on a new line as well
```
your code here

```

11

Thanks, sterretje! I edited my post again to follow your directions. I copied your three back ticks and put them at the beginning and at the end of the code, each set on their own line. Then the code magically appeared in the code box (with its own scroll bar) as it used to do in the old Forum. You're a genius, and you were kind enough to make me look a little less of an idiot. Much appreciated.

1 Like
12

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.