Reading from CSV file and storing into multiple arrays with SdFat library

Hi! I’m working on a project that requires me to read a CSV file from an SD Card, then store it into several arrays. The file contains 300 floats and 4 ints. The floats need to be stored into three separate arrays, each of size 100. I’ve managed to make it store all of the points in the arrays to the SD Card, but I’m having issues reading the file it creates. I’m using the SDFat library with an Arduino Uno to do all of this.

Here is my code for storing the data:

void save()
{    
  char filename[] = "ROUTIN00.CSV";             // Create a new file
  for (filecount = 0; filecount < 100; filecount++) 
  {
    filename[6] = filecount/10 + '0';
    filename[7] = filecount%10 + '0';
    if (!sd.exists(filename)) 
    {               
      routinefile.open(filename);
      Serial.println(filename); 
      break;                                    
    }
    else Serial.println(F("next file"));
  }    
  if (!routinefile) 
  {
    Serial.println(F("Couldn't create file"));
  }
  for(xyzpoints = 1; xyzpoints < 101; xyzpoints++)
  {
    routinefile << x[xyzpoints] << "," << y[xyzpoints] << "," << z[xyzpoints] << "," << endl;
  }
  routinefile << stores << "," <<  count << "," << rows << "," << columns << "," << endl << flush;
}

Here is the code I’ve currently got to load a file:
void load()

{
  fileselect[6] = filenum/10 + '0';
  fileselect[7] = filenum%10 + '0';
  Serial.println(fileselect);
  ifstream sdin(fileselect);
  xyzpoints = 1;
  for(xyzpoints = 1; xyzpoints < 101; xyzpoints++)
  {
    sdin >> x[xyzpoints] >> y[xyzpoints] >> z[xyzpoints];
  }
  sdin >> stores >> count >> rows >> columns;
}

(fileselect is in the form ROUTIN00.CSV, and the value of filenum depends on which file is selected in the menu, which is stored on a Gameduino 2).

The save command works perfectly. To demonstrate what’s happening, here’s an example of a file I saved. The first column is the x array, the middle column is the y array, and the last column is the z array. The last four ints are stores, count, rows, and columns, respectively. This is the saved file:

1120,538,0,
1556,695,0,
2197,1050,0,
3252,1037,0,
3510,926,0,
3657,705,0,
4139,658,0,
4266,1026,0,
4494,1505,0,
5186,1516,0,
6045,2167,0,
6865,2386,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
0,0,0,
13,12,1,1

This is what happens when I try to read this file:

1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1120.00,1120.00,1120.00,
1,0,1,1,

As you can see, the first x value is the only value that is read, and fills the entire x, y, and z arrays. The last 4 values are ignored entirely (1,0,1,1 are the default values).

I’ve been working on this for a while now, and have done some extensive searching for an answer, but I can’t quite get this to work. I think the problem is probably my not really understanding the ifstream class, which I got from the readCSV example in the SdFat library. If anyone could point me in the right direction, it’d be greatly appreciated.

(I apologize if I’ve made any egregiously stupid errors, I’ve only been working with this for a month or so and didn’t have any prior programming experience.)

First, you should check for errors when you read from a file like this:

  if (!(sdin >> x[xyzpoints] >> y[xyzpoints] >> z[xyzpoints])) {
    // Handle read error.
  }

Second, you must provide a char variable to read the comma separators:

  if (!(sdin >> x[xyzpoints] >> sep1 >> y[xyzpoints] >> sep >> z[xyzpoints]) || sep1 != ',' || sep2 != ',') {
    // Handle read error.
  }

Where sep1 and sep2 are type char:

  char sep1, sep2;

Thanks for the speedy response! I tried your suggestion and am now significantly closer to what I want to do, but I’ve still got a small issue somewhere. Here’s my new code:

void load()
{
  fileselect[6] = filenum/10 + '0';
  fileselect[7] = filenum%10 + '0';
  Serial.println(fileselect);
  xyzpoints = 0;
  ifstream sdin(fileselect);
  for(xyzpoints = 1; xyzpoints < 101; xyzpoints++)
  {
    char sep1, sep2;
    if (!(sdin >> x[xyzpoints] >> sep1 >> y[xyzpoints] >> sep2 >> z[xyzpoints]) || sep1 != ',' || sep2 != ',') 
    {
      Serial.print(F("Read Error on point "));
      Serial.print(xyzpoints);
      Serial.print(F("   -   point = ["));
      Serial.print(x[xyzpoints]);
      Serial.print(F(", "));
      Serial.print(y[xyzpoints]);
      Serial.print(F(", "));
      Serial.print(z[xyzpoints]);
      Serial.print(F("]   -   Sep = "));
      Serial.print(sep1);
      Serial.print(F(" "));
      Serial.println(sep2);
    }
    else
    {
      Serial.print(F("Works for point "));
      Serial.print(xyzpoints);
      Serial.print(F("   -   point = ["));
      Serial.print(x[xyzpoints]);
      Serial.print(F(", "));
      Serial.print(y[xyzpoints]);
      Serial.print(F(", "));
      Serial.print(z[xyzpoints]);
      Serial.print(F("]   -   Sep = "));
      Serial.print(sep1);
      Serial.print(F(" "));
      Serial.println(sep2);
    }
  }
  char sep1, sep2, sep3;
  if (!(sdin >> stores >> sep1 >> count >> sep2 >> rows >> sep3 >> columns) || sep1 != ',' || sep2 != ',' || sep3 != ',') 
  {
    Serial.print(F("Read Error on final line"));
    Serial.print(F("   -   values = ["));
    Serial.print(stores);
    Serial.print(F(", "));
    Serial.print(count);
    Serial.print(F(", "));
    Serial.print(rows);
    Serial.print(F(", "));
    Serial.print(columns);
    Serial.print(F("]   -   Sep = "));
    Serial.print(sep1);
    Serial.print(F(" "));
    Serial.print(sep2);
    Serial.print(F(" "));
    Serial.println(sep3);
  }
  else
  {
    Serial.print(F("Read Error on final line"));
    Serial.print(F("   -   values = ["));
    Serial.print(stores);
    Serial.print(F(", "));
    Serial.print(count);
    Serial.print(F(", "));
    Serial.print(rows);
    Serial.print(F(", "));
    Serial.print(columns);
    Serial.print(F("]   -   Sep = "));
    Serial.print(sep1);
    Serial.print(F(" "));
    Serial.print(sep2);
    Serial.print(F(" "));
    Serial.println(sep3);
  }
}

And here’s my Serial monitor when I try to load the same file as before:

Initializing SD card...initialization done.
ROUTIN05.CSV
Works for point 1   -   point = [1120.00, 538.00, 0.00]   -   Sep = , ,
Read Error on point 2   -   point = [0.00, 1556.00, 695.00]   -   Sep = , ,
Read Error on point 3   -   point = [695.00, 0.00, 2197.00]   -   Sep = , ,
Read Error on point 4   -   point = [2197.00, 1050.00, 0.00]   -   Sep = , ,
Read Error on point 5   -   point = [0.00, 3252.00, 1037.00]   -   Sep = , ,
Read Error on point 6   -   point = [1037.00, 0.00, 3510.00]   -   Sep = , ,
Read Error on point 7   -   point = [3510.00, 926.00, 0.00]   -   Sep = , ,
Read Error on point 8   -   point = [0.00, 3657.00, 705.00]   -   Sep = , ,
Read Error on point 9   -   point = [705.00, 0.00, 4139.00]   -   Sep = , ,
Read Error on point 10   -   point = [4139.00, 658.00, 0.00]   -   Sep = , ,
Read Error on point 11   -   point = [0.00, 4266.00, 1026.00]   -   Sep = , ,
Read Error on point 12   -   point = [1026.00, 0.00, 4494.00]   -   Sep = , ,
Read Error on point 13   -   point = [4494.00, 1505.00, 0.00]   -   Sep = , ,
Read Error on point 14   -   point = [0.00, 5186.00, 1516.00]   -   Sep = , ,
Read Error on point 15   -   point = [1516.00, 0.00, 6045.00]   -   Sep = , ,
Read Error on point 16   -   point = [6045.00, 2167.00, 0.00]   -   Sep = , ,
Read Error on point 17   -   point = [0.00, 6865.00, 2386.00]   -   Sep = , ,
Read Error on point 18   -   point = [2386.00, 0.00, 0.00]   -   Sep = , ,
Read Error on point 19   -   point = [0.00, 0.00, 0.00]   -   Sep = , ,
Read Error on point 20   -   point = [0.00, 0.00, 0.00]   -   Sep = , ,
(it's the same for every point in between these lines, so I've cut them out to save space)
Read Error on point 99   -   point = [0.00, 0.00, 0.00]   -   Sep = , ,
Read Error on point 100   -   point = [0.00, 0.00, 0.00]   -   Sep = , ,
Read Error on final line   -   values = [1, 0, 0, 0]   -   Sep = , , ,

It appears to be copying the previous line’s z value into each line’s x value. I’ve been trying to track down the source of this issue without much success yet. Thanks again for your assistance, it is much appreciated!

Sorry, I didn’t notice that you have an extra comma at the end of input lines. Either remove the extra comma in the input or add a third char variable to read this comma.

1120,538,0, <-- extra comma
1556,695,0, <-- extra comma

It appears to be copying the previous line’s z value into each line’s x value. I’ve been trying to track down the source of this issue without much success yet.

Trying to read a comma as a number causes an error and leaves the file’s position to be incorrect.

You must provide a variable for every item in the input line. White space is skipped so you don’t need to read the CR/LF at the end of lines.

Sure enough, that fixed the issue and the load function now works perfectly. Thanks very much for your help and continued support for this library!