Arduino Forum

Using Arduino => Storage => Topic started by: Duke1215 on Apr 07, 2014, 11:11 pm

Title: Read CSV or TXT from SD card into string or array
Post by: Duke1215 on Apr 07, 2014, 11:11 pm
I am working on an system and need to read values on an sd card. The txt or csv will have multiple lines with 2 rows of values. What I would like to achieve is for the arduino to open the sd file, take line one, split it into its two values. val1 and val2. It will then compare the values to what a sensor reads. If they match, it will stop searching the txt file. If the values don't match, it will read and compare the values from the next line in the txt file. I am very new to programming and think the values would either be a string or array. Please help! Thanks
Title: Re: Read CSV or TXT from SD card into string or array
Post by: CatweazleNZ on Apr 15, 2014, 11:33 am
You can read SD card files line-by-line if this helps:

Code: [Select]
String l_line = "";
//open the file here
while (l_SDFile.available() != 0) {
    //A inconsistent line length may lead to heap memory fragmentation
    l_line = l_SDFile.readStringUntil('\n');
    if (l_line == "") //no blank lines are anticipated
      break;
    //

   //parse l_line here
}
//close the file here

   

Cheers

Catweazle NZ
Title: Re: Read CSV or TXT from SD card into string or array
Post by: CatweazleNZ on Apr 15, 2014, 11:38 am
Hi

This more substantial procedure loads a backup file with multiple record types into memory arrays:

Code: [Select]


boolean LoadRecentClimateBackupFile(){
 const byte c_proc_num = 31;
 Push(c_proc_num);
 //After Climate Arrays have been initialised (nulled)
 //After call to CalcNextClimateCheck()

/*    
  30MN Left Temp 22 22 22 23 23 23 23 23 24 24 24 24
  30MN Left Humidity 62 62 62 60 60 60 60 59 59 59 59 59
  DAY Left Temp Max NULL NULL NULL NULL NULL 24 24
  DAY Left Temp Min NULL NULL NULL NULL NULL 22 20
  DAY Left Humidity Max NULL NULL NULL NULL NULL 67 62
  DAY Left Humidity Min NULL NULL NULL NULL NULL 58 59
  AIRPUMP 22 22 22 23 23 23 23 23 24 24 24 24
*/

 if (!G_SDCardOK) {
   Pop(c_proc_num);
   return false;
 }

 SPIDeviceSelect(DC_SDCardSSPin);

 int l_yyyy, l_mm, l_dd, l_hh, l_mn = 0; //l_ss
 String l_SD_filename = "";

 long l_PeriodTime   = 0;
 int  l_PeriodOffset = 0;
 boolean l_SD_file_found = false;
 //Look back up to 12 climate periods for the most recent backup file
 for (byte l_count = 1; l_count <= 12 ; l_count++) {
   l_PeriodTime = Date() + G_NextClimateUpdateTime - long(100000 * l_count / G_Periods_Per_Day) + 1; //Fourth backup ago, fraction adjusted
   DateDecode(l_PeriodTime, &l_dd, &l_mm, &l_yyyy);
   TimeDecode(l_PeriodTime, &l_hh, &l_mn);
   //l_SD_filename = String(ZeroFill(l_mm,2) + ZeroFill(l_dd,2) + ZeroFill(l_hh,2) + ZeroFill(l_mn,2) + ".TXT");
   String l_directory = String(EPSR(E_BACKUPSsl_2217) + ZeroFill(l_yyyy,4) + "/" + ZeroFill(l_mm,2) + "/" + ZeroFill(l_dd,2) );
   SD.mkdir(&l_directory[0]);
   l_SD_filename = String(l_directory + "/" + ZeroFill(l_mm,2) + ZeroFill(l_dd,2) + ZeroFill(l_hh,2) + ZeroFill(l_mn,2) + ".txt");
   //BACKUPS/2013/11/10/11101800.txt
   if (SD.exists(&l_SD_filename[0])) {
     l_SD_file_found = true;
     break;
   }
   l_PeriodOffset += 1;
 }
 CheckRAM();
 if (!l_SD_file_found) {
   SPIDeviceSelect(DC_EthernetSSPin);
   Pop(c_proc_num);
   return false;
 }

 int l_DayOffset = 0;
 //If the date of the backup file we will load is yesterday
 //we need to offset (7) DAY climate data by one day.
 if (l_PeriodTime < Date()) //Date() is midnight 12:00am - the start of today
   l_DayOffset = 1;
 //

 //We have found the most recent backup and have offsets for missed data
 //BACKUPS/2013/11/10/11101800.TXT
 //Serial.println(l_SD_filename);
 File l_SD_file = SD.open(&l_SD_filename[0],FILE_READ);
 CheckRAM();
 if (! l_SD_file) {
   ActivityWrite(String(EPSR(E_Could_not_open_file__1345) + l_SD_filename)); //Should not happen - we know it exists
   SPIDeviceSelect(DC_EthernetSSPin);
   Pop(c_proc_num);
   return false;
 }

 int* l_row; //An int array pointer
 String l_reqd_stattype = DetermineStatType(G_Periods_Per_Day);
 while (l_SD_file.available() != 0) {
   //Hopefully the relatively small size of the files and consistent
   //line length will minimise any String memory fragmentation
   String l_line = l_SD_file.readStringUntil('\n');
   if (l_line == "") //no blank lines are anticipated
     break;
   //
   //Serial.println(l_line);
   //30MN Right Humidity 60 60 60 60 60 59 58 58 58 58 58 58
   //DAY Left Temp Max NULL NULL NULL NULL NULL 24 24
   //AIRPUMP 22 22 22 23 23 23 23 23 24 24 24 24

   int l_posn = 0;
   String l_stattype = ENDF2(l_line,l_posn,'\t');
   if (l_stattype == "AIRPUMP") {
     int l_index1  = DC_ClimatePeriodCount - 1;
     int l_Offset = l_PeriodOffset;
     for (int l_count2 = 1; l_count2 <= DC_ClimatePeriodCount; l_count2++) { //twelve
       String l_value = ENDF2(l_line,l_posn,'\t');
       if (l_value == "") //If index not back to zero we have gaps in the recovered data
                          //because we could not load the most recent backup file.
         break; //we have finished
       //
       if (l_Offset == 0) { //We can load this value
         if (l_value != EPSR(E_NULL_2006))
           G_AirPumpModes[l_index1] = l_value.toInt();
         else
           G_AirPumpModes[l_index1] = 0;
         //
         l_index1--;
       }
       else //We discard end value(s) because we only have an old backup (not the most recent)
         l_Offset--; //By skipping index values we will leave null gaps in the data in the lower array index values
       //
       if (l_index1 < 0)
         break;
       //
     } //for each array element
   }
   else {
     String l_sensor   = ENDF2(l_line,l_posn,'\t');
     String l_stat     = ENDF2(l_line,l_posn,'\t');
     int l_sensor_index = 0; //scoped for later use
     boolean l_sensor_found = false;
     for (l_sensor_index = 0; l_sensor_index < DC_SensorCountTemp; l_sensor_index++) {
       if (C_SensorList[l_sensor_index] == l_sensor) {
         l_sensor_found = true;  
         break;
       }
       //
     } //find the sensor
     if (l_sensor_found) { //skip removed sensors
       if (l_reqd_stattype == l_stattype) {
         //This is a periodic data set
         int l_ValueOffset = 0;
         if (l_stat == EPSR(E_Temp_2398)) {
           l_row = G_ArrayTemp[l_sensor_index].TempPeriodic;
           l_ValueOffset = DC_Temp_UpShift;
         }
         else if (l_sensor_index < DC_SensorCountHum) //Humidity
           l_row = G_ArrayHum[l_sensor_index].HumPeriodic;
         //
 
         if ((l_stat == EPSR(E_Temp_2398)) || (l_sensor_index < DC_SensorCountHum)) {
           int l_index1  = DC_ClimatePeriodCount - 1;
           int l_Offset = l_PeriodOffset;
           for (int l_count2 = 1; l_count2 <= DC_ClimatePeriodCount; l_count2++) { //twelve
             String l_value = ENDF2(l_line,l_posn,'\t');
             if (l_value == "") //If index not back to zero we have gaps in the recovered data
               //because we could not load the most recent backup file.
               break; //we have finished
             //
             if (l_Offset == 0) { //We can load this value
               if (l_value != EPSR(E_NULL_2006))
                 l_row[l_index1] = int((StringToDouble(l_value) + l_ValueOffset) * 10);
               else
                 l_row[l_index1] = 0;
               //
               l_index1--;
             }
             else //We discard end value(s) because we only have an old backup (not the most recent)
               l_Offset--; //By skipping index values we will leave null gaps in the data in the lower array index values
             //
             if (l_index1 < 0)
               break;
             //
           } //for each array element
         } //skip humidity statistics not now required
       } //periodic stat type
       else if (l_stattype == "DAY") {
         String l_sub_stat = ENDF2(l_line,l_posn,'\t'); //Min or Max
         //This is a day min/max data set
         int l_ValueOffset = 0;
         l_row = NULL;
         if ((l_stat == EPSR(E_Temp_2398)) && (l_sub_stat == "Max")) {
           l_row = G_ArrayTemp[l_sensor_index].TempMax;
           l_ValueOffset = DC_Temp_UpShift;
         }
         else if ((l_stat == EPSR(E_Temp_2398)) && (l_sub_stat == "Min")) {
           l_row = G_ArrayTemp[l_sensor_index].TempMin;
           l_ValueOffset = DC_Temp_UpShift;
         }
         else if ((l_stat == EPSR(E_Humidity_1184)) && (l_sub_stat == "Max")) {
           if (l_sensor_index  < DC_SensorCountHum)
             l_row = G_ArrayHum[l_sensor_index].HumMax;
         }
         else { //if ((l_stat == "Humidity") && (l_sub_stat == "Min"))
           if (l_sensor_index  < DC_SensorCountHum)
             l_row = G_ArrayHum[l_sensor_index].HumMin;
         }
         //
         if ((l_stat == EPSR(E_Temp_2398)) || (l_sensor_index < DC_SensorCountHum)) {
           int l_index2 = DC_WeekDayCount - 1;
           int l_Offset = l_DayOffset;
           for (int l_count3 = 1; l_count3 <= DC_WeekDayCount; l_count3++) { //7 days
             String l_value = ENDF2(l_line,l_posn,'\t');
             if (l_value == "")
               break; //we have finished
             //
             if (l_Offset == 0) {
               if (l_value != EPSR(E_NULL_2006))
                 l_row[l_index2] = int((StringToDouble(l_value) + l_ValueOffset) * 10);
               else
                 l_row[l_index2] = 0;
               //
               l_index2--;
             }
             else
               l_Offset--;
             //
             if (l_index2 < 0)
               break;
             //
           } //Load each day of the week
         } //skip humidity statistics not now required
       } //Determine what type of statistic (Periodic or DAY)
     } //Skip removed sensors
   } //Process AIRPUMP data differently
 } //EOF
 CheckRAM();
 l_SD_file.close();

 SPIDeviceSelect(DC_EthernetSSPin);

 Pop(c_proc_num);
 return true;
} //proc


Catweazle NZ
Title: Re: Read CSV or TXT from SD card into string or array
Post by: CatweazleNZ on Apr 15, 2014, 11:40 am
You will also need this to parse your input lines if you are using a delimiter between multiple fields/values on each line:

Code: [Select]

String ENDF2(String &p_line, int &p_start, char p_delimiter) {
  const byte c_proc_num = 27;
  Push(c_proc_num);
//Extract fields from a line one at a time based on a delimiter.
//Because the line remains intact we dont fragment heap memory
//p_start would normally start as 0
//p_start increments as we more along the line
//We return p_start = -1 with the last field

  //If we have already parsed the whole line then return null
  if (p_start == -1) {
Pop(c_proc_num);
return "";
  }

  CheckRAM();
  int l_start = p_start;
  int l_index = p_line.indexOf(p_delimiter,l_start);
  if (l_index == -1) { //last field of the data line
p_start = l_index;
Pop(c_proc_num);
return p_line.substring(l_start);
  }
  else { //take the next field off the data line
p_start = l_index + 1;
Pop(c_proc_num);
return p_line.substring(l_start,l_index); //Include, Exclude
  }
  //Pop(c_proc_num);
}


Ignore calls to Push, Pop and CheckRAM.

Cheers

Catweazle NZ
Title: Re: Read CSV or TXT from SD card into string or array
Post by: fat16lib on Apr 15, 2014, 01:21 pm
Here is a simple sketch that reads a CSV file with two numbers on each line.  It doesn't use the dangerous String class.
Code: [Select]

#include <SD.h>
File file;

bool readLine(File &f, char* line, size_t maxLen) {
  for (size_t n = 0; n < maxLen; n++) {
    int c = f.read();
    if ( c < 0 && n == 0) return false;  // EOF
    if (c < 0 || c == '\n') {
      line[n] = 0;
      return true;
    }
    line[n] = c;
  }
  return false; // line too long
}

bool readVals(long* v1, long* v2) {
  char line[40], *ptr, *str;
  if (!readLine(file, line, sizeof(line))) {
    return false;  // EOF or too long
  }
  *v1 = strtol(line, &ptr, 10);
  if (ptr == line) return false;  // bad number if equal
  while (*ptr) {
    if (*ptr++ == ',') break;
  }
  *v2 = strtol(ptr, &str, 10);
  return str != ptr;  // true if number found
}

void setup(){
  long x, y;
  Serial.begin(9600);
  if (!SD.begin(SS)) {
    Serial.println("begin error");
    return;
  }
  file = SD.open("TEST.CSV", FILE_READ);
  if (!file) {
    Serial.println("open error");
    return;
  }
  while (readVals(&x, &y)) {
    Serial.print("x: ");
    Serial.println(x);
    Serial.print("y: ");
    Serial.println(y);
    Serial.println();
  }
  Serial.println("Done");
}
void loop() {}
 

Reading this "TEST.CSV" file.
Quote

1,2
34,56
987,-765
1234,-5678


Results in this output
Quote

x: 1
y: 2

x: 34
y: 56

x: 987
y: -765

x: 1234
y: -5678

Done
Title: Re: Read CSV or TXT from SD card into string or array
Post by: orthho on Mar 16, 2015, 09:08 am
Hi fat16lib..i want to ask some question the same that i've posted in other topics
http://forum.arduino.cc/index.php?topic=127711.15 (http://forum.arduino.cc/index.php?topic=127711.15) (i didn't mean to flood or something)

Here is a simple sketch that reads a CSV file with two numbers on each line.  It doesn't use the dangerous String class.
Code: [Select]

#include <SD.h>
File file;

bool readLine(File &f, char* line, size_t maxLen) {
  for (size_t n = 0; n < maxLen; n++) {
    int c = f.read();
    if ( c < 0 && n == 0) return false;  // EOF
    if (c < 0 || c == '\n') {
      line[n] = 0;
      return true;
    }
    line[n] = c;
  }
  return false; // line too long
}

bool readVals(long* v1, long* v2) {
  char line[40], *ptr, *str;
  if (!readLine(file, line, sizeof(line))) {
    return false;  // EOF or too long
  }
  *v1 = strtol(line, &ptr, 10);
  if (ptr == line) return false;  // bad number if equal
  while (*ptr) {
    if (*ptr++ == ',') break;
  }
  *v2 = strtol(ptr, &str, 10);
  return str != ptr;  // true if number found
}

void setup(){
  long x, y;
  Serial.begin(9600);
  if (!SD.begin(SS)) {
    Serial.println("begin error");
    return;
  }
  file = SD.open("TEST.CSV", FILE_READ);
  if (!file) {
    Serial.println("open error");
    return;
  }
  while (readVals(&x, &y)) {
    Serial.print("x: ");
    Serial.println(x);
    Serial.print("y: ");
    Serial.println(y);
    Serial.println();
  }
  Serial.println("Done");
}
void loop() {}
 

Reading this "TEST.CSV" file.
Results in this output
thanks that you've already post coding to read value from .CSV.. but I'm hardly to understand, because i'm new to arduino and C language..
could you give explanation..sorry if i'm really burdensome... :(
Title: Re: Read CSV or TXT from SD card into string or array
Post by: abinaustin on Jul 26, 2015, 08:41 pm
HI fat16lib
CAN YOU EXPLAIN HOW DID YOU USE POINTERS TO READ VALUE FROM SD CARD.ESPECIIALY*PTR ,IS THAT DEFAULT POINTER? I DONT UNDERSTAND THE CODE. CAN YOU BREIFLY EXPLIAN
THANKYOU
Title: reading csv file ,easy logic
Post by: abinaustin on Aug 01, 2015, 08:07 am
// easy program that reads csv file.


#include <SPI.h>
#include <SD.h>

File myFile;

void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }


  Serial.print("Initializing SD card...");

  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");

  // re-open the file for reading:
  myFile = SD.open("TEST.CSV");
  if (myFile) {
    Serial.println("test.txt:");

    // read from the file until there's nothing else in it:
    while (myFile.available()) {
      String a="";
      for(int i=0;i<9;++i)
      {
      int j;
       char temp=myFile.read();
        if(temp!=','&&temp!='\n')
      { //a=temp;
        a+=temp;}
      else if(temp==','||temp=='\n'){
        j=a.toInt();
     // Serial.println(a);
      Serial.println(j);
      break;}
     
        }
     
    }
    // close the file:
    myFile.close();
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void loop()
{
  // nothing happens after setup
}






/*
reads the values from file as string then converts to int, the ',' is used to seperate it .*/
Title: Re: Read CSV or TXT from SD card into string or array
Post by: CatweazleNZ on Aug 08, 2015, 09:54 am
Hi

I cannot say I like this as an example of a generic way to read CSV text files:
Code: [Select]
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
      String a="";
      for(int i=0;i<9;++i) {
        int j;
        char temp=myFile.read();
        if(temp!=','&&temp!='\n') {
          a+=temp;}
        else //if(temp==','||temp=='\n'){
          j=a.toInt();
          Serial.println(j);
          break;
        }
      } //Only read eight characters from one line
    } //while (myFile.available())
    // close the file:


These two lines may fragment heap memory with tiny unused old strings - one 1 byte, one 2 bytes, etc.

Code: [Select]

      String a="";

      a+=temp;



And this seems an unnecessary way to process individual lines - also each line cannot be longer that eight bytes.
Code: [Select]
     
for(int i=0;i<9;++i) {
...
}



This untested code might be a better generic algorithm for loading of CSV text files:
Code: [Select]
//Easy program that reads a csv file
//Parsing comma separated fields
//We assume all the fields on each line are integers
#include <SPI.h>
#include <SD.h>

void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }

  Serial.print("Initializing SD card...");
  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");

  // re-open the file for reading:
  File myFile = SD.open("TEST.CSV");
  if (!myFile) {
    Serial.println("error opening test.txt");
    return;
  }

  Serial.println("test.txt:");

  // read from the file until there's nothing else in it:
  String l_line;
  l_line.reserve(128); //Avoids heap memory fragmentation
                       //Reserve space for your longest expected data line
  while (myFile.available()) {
    l_line = myFile.readStringUntil('\n');
    l_line.trim();
    if (l_line != "") {
      int l_start_posn = 0;
      while (l_start_posn != -1)
        Serial.println(ENDF2(l_line,l_start_posn,',').toInt());
      //
    } //skip blank (NULL) lines
  }//Read the file line by line
  myFile.close();
   
} //setup

void loop()
{
  // nothing happens after setup
}

String ENDF2(String &p_line, int &p_start, char p_delimiter) {
//EXTRACT NEXT DELIMITED FIELD VERSION 2
//Extract fields from a line one at a time based on a delimiter.
//Because the line remains intact we dont fragment heap memory
//p_start would normally start as 0
//p_start increments as we move along the line
//We return p_start = -1 with the last field

  //If we have already parsed the whole line then return null
  if (p_start == -1) {
    return "";
  }

  int l_start = p_start;
  int l_index = p_line.indexOf(p_delimiter,l_start);
  if (l_index == -1) { //last field of the data line
    p_start = l_index;
    return p_line.substring(l_start);
  }
  else { //take the next field off the data line
    p_start = l_index + 1;
    return p_line.substring(l_start,l_index); //Include, Exclude
  }
}


Cheers

Catweazle NZ
Title: Re: Read CSV or TXT from SD card into string or array
Post by: fat16lib on Aug 09, 2015, 06:12 pm
Here (http://forum.arduino.cc/index.php?topic=340849.msg2349945) is a simple function for reading CSV files.
Title: Re: Read CSV or TXT from SD card into string or array
Post by: rkn2008 on Nov 15, 2015, 07:18 pm
Here is a simple sketch that reads a CSV file with two numbers on each line.  It doesn't use the dangerous String class.
Code: [Select]

#include <SD.h>
File file;

bool readLine(File &f, char* line, size_t maxLen) {
  for (size_t n = 0; n < maxLen; n++) {
    int c = f.read();
    if ( c < 0 && n == 0) return false;  // EOF
    if (c < 0 || c == '\n') {
      line[n] = 0;
      return true;
    }
    line[n] = c;
  }
  return false; // line too long
}

bool readVals(long* v1, long* v2) {
  char line[40], *ptr, *str;
  if (!readLine(file, line, sizeof(line))) {
    return false;  // EOF or too long
  }
  *v1 = strtol(line, &ptr, 10);
  if (ptr == line) return false;  // bad number if equal
  while (*ptr) {
    if (*ptr++ == ',') break;
  }
  *v2 = strtol(ptr, &str, 10);
  return str != ptr;  // true if number found
}

void setup(){
  long x, y;
  Serial.begin(9600);
  if (!SD.begin(SS)) {
    Serial.println("begin error");
    return;
  }
  file = SD.open("TEST.CSV", FILE_READ);
  if (!file) {
    Serial.println("open error");
    return;
  }
  while (readVals(&x, &y)) {
    Serial.print("x: ");
    Serial.println(x);
    Serial.print("y: ");
    Serial.println(y);
    Serial.println();
  }
  Serial.println("Done");
}
void loop() {}
 

Reading this "TEST.CSV" file.
Results in this output
This is exactly what I need but I have 3 variables.
How could I modify this to handle 3 or more variable???
Title: Re: Read CSV or TXT from SD card into string or array
Post by: fat16lib on Nov 15, 2015, 09:56 pm
Here is an example that reads a CSV file into an array with any number of columns/rows.
Code: [Select]

// Example to read a two dimensional array.
//
#include <SPI.h>
#include <SD.h>
#define CS_PIN 10

// 5 X 4 array
#define ROW_DIM 5
#define COL_DIM 4

File file;

/*
 * Read a file one field at a time.
 *
 * file - File to read.
 *
 * str - Character array for the field.
 *
 * size - Size of str array.
 *
 * delim - String containing field delimiters.
 *
 * return - length of field including terminating delimiter.
 *
 * Note, the last character of str will not be a delimiter if
 * a read error occurs, the field is too long, or the file
 * does not end with a delimiter.  Consider this an error
 * if not at end-of-file.
 *
 */
size_t readField(File* file, char* str, size_t size, char* delim) {
  char ch;
  size_t n = 0;
  while ((n + 1) < size && file->read(&ch, 1) == 1) {
    // Delete CR.
    if (ch == '\r') {
      continue;
    }
    str[n++] = ch;
    if (strchr(delim, ch)) {
        break;
    }
  }
  str[n] = '\0';
  return n;
}
//------------------------------------------------------------------------------
#define errorHalt(msg) {Serial.println(F(msg)); while(1);}
//------------------------------------------------------------------------------
void setup() {
  Serial.begin(9600);

  // Initialize the SD.
  if (!SD.begin(CS_PIN)) {
    errorHalt("begin failed");
  }
  // Create or open the file.
  file = SD.open("READNUM.TXT", FILE_WRITE);
  if (!file) {
    errorHalt("open failed");
  }
  // Rewind file so test data is not appended.
  file.seek(0);

  // Write test data.
  file.print(F(
    "11,12,13,14\r\n"
    "21,22,23,24\r\n"
    "31,32,33,34\r\n"
    "41,42,43,44\r\n"
    "51,52,53,54"     // Allow missing endl at eof.
    ));

  // Rewind the file for read.
  file.seek(0);

  // Array for data.
  int array[ROW_DIM][COL_DIM];
  int i = 0;     // First array index.
  int j = 0;     // Second array index
  size_t n;      // Length of returned field with delimiter.
  char str[20];  // Must hold longest field with delimiter and zero byte.
  char *ptr;     // Test for valid field.

  // Read the file and store the data.
 
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      n = readField(&file, str, sizeof(str), ",\n");
      if (n == 0) {
        errorHalt("Too few lines");
      }
      array[i][j] = strtol(str, &ptr, 10);
      if (ptr == str) {
        errorHalt("bad number");
      }
      while (*ptr == ' ') {
        ptr++;
      }
      if (*ptr != ',' && *ptr != '\n' && *ptr != '\0') {
        errorHalt("extra characters in field");
      }
      if (j < (COL_DIM-1) && str[n-1] != ',') {
        errorHalt("line with too few fields");
      }
    }
    // Allow missing endl at eof.
    if (str[n-1] != '\n' && file.available()) {
      errorHalt("missing endl");
    }   
  }

  // Print the array.
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      if (j) {
        Serial.print(' ');
      }
      Serial.print(array[i][j]);
    }
    Serial.println();
  }
  Serial.println("Done");
  file.close();
}
//------------------------------------------------------------------------------
void loop() {
}

Title: Re: Read CSV or TXT from SD card into string or array
Post by: rkn2008 on Nov 16, 2015, 03:01 am
im sorry i just cant follow.

I can be clearer i guess or maybe im just retarded.

I have a CSV file with 3 pieces of data.
with an ever changing amount of rows.  The columns will always be 3 (date_time, lat, lon)

I need to read that file from LOG.CSV to the variables of date_time lat and lon so i can put them in a POST statement and have them sent to a PHP web-server as:
log.php?date_time=date_time&lat=lat&lon=lon

Does this make sense??

Please let me know what you think.  i am just stuck in this rut.
Title: Re: Read CSV or TXT from SD card into string or array
Post by: NewbieNewbie on Jan 31, 2016, 09:08 pm
Thanks for multirow, multicolumn read code.

I am trying to modify so that it ignores the condition of too few fields.  I have designed file data structure to have variable columns. 

I will continue to try and hope to get it working before you reply.
Title: Re: Read CSV or TXT from SD card into string or array
Post by: xiaoyan1994 on Apr 06, 2016, 05:24 pm
Hi fat16lib. How can I read six decimals places from files instead of integrals by using your code?
Title: Re: Read CSV or TXT from SD card into string or array
Post by: fat16lib on Apr 06, 2016, 08:25 pm
Use a floating type for the variable and use strtod() in place of strtol();

Code: [Select]

  //  int array[ROW_DIM][COL_DIM];

  //  change array type to float or double (both are 32-bit floating point on AVR)
  double array[ROW_DIM][COL_DIM];

  // ...

  //  array[i][j] = strtol(str, &ptr, 10);

  //  convert filed to floating point instead of long.
      array[i][j] = strtod(str, &ptr);

Title: Re: Read CSV or TXT from SD card into string or array
Post by: hiramvillarreal on Jun 06, 2016, 03:26 pm
Hi fat16lib, great solution to read a CSV, I need to output almost the same but I need to output a line once and never output it again.

I am using a CSV to load a list of "user" and "password" from a PC, then insert the SD card into the Arduino card reader, after push a bottom it will output on a LCD an unused line, the next time the push bottom it will press, it will give the next line.

I been thinking  to use an extra file to save the last position read it and stop the while loop keeping that X,Y value for the display, after that the "index" file will be saved with an increment of the last position.
Do you have an advice to do so?

Thanks  sorry for my English.
Title: Re: Read CSV or TXT from SD card into string or array
Post by: pussimus on Feb 20, 2017, 04:01 pm
Here is a simple sketch that reads a CSV file with two numbers on each line.  It doesn't use the dangerous String class.
Code: [Select]

#include <SD.h>
File file;

bool readLine(File &f, char* line, size_t maxLen) {
  for (size_t n = 0; n < maxLen; n++) {
    int c = f.read();
    if ( c < 0 && n == 0) return false;  // EOF
    if (c < 0 || c == '\n') {
      line[n] = 0;
      return true;
    }
    line[n] = c;
  }
  return false; // line too long
}

bool readVals(long* v1, long* v2) {
  char line[40], *ptr, *str;
  if (!readLine(file, line, sizeof(line))) {
    return false;  // EOF or too long
  }
  *v1 = strtol(line, &ptr, 10);
  if (ptr == line) return false;  // bad number if equal
  while (*ptr) {
    if (*ptr++ == ',') break;
  }
  *v2 = strtol(ptr, &str, 10);
  return str != ptr;  // true if number found
}

void setup(){
  long x, y;
  Serial.begin(9600);
  if (!SD.begin(SS)) {
    Serial.println("begin error");
    return;
  }
  file = SD.open("TEST.CSV", FILE_READ);
  if (!file) {
    Serial.println("open error");
    return;
  }
  while (readVals(&x, &y)) {
    Serial.print("x: ");
    Serial.println(x);
    Serial.print("y: ");
    Serial.println(y);
    Serial.println();
  }
  Serial.println("Done");
}
void loop() {}
 
Is it just me, or doesnt it work anymore ?
I updated Library SD.H (v1.1.1) after that it wont read files from my SD card.
Title: Re: Read CSV or TXT from SD card into string or array
Post by: MikeOrlando02 on Jun 19, 2019, 03:02 pm
Here is a simple sketch that reads a CSV file with two numbers on each line.  It doesn't use the dangerous String class.
Code: [Select]

#include <SD.h>
File file;

bool readLine(File &f, char* line, size_t maxLen) {
  for (size_t n = 0; n < maxLen; n++) {
    int c = f.read();
    if ( c < 0 && n == 0) return false;  // EOF
    if (c < 0 || c == '\n') {
      line[n] = 0;
      return true;
    }
    line[n] = c;
  }
  return false; // line too long
}

bool readVals(long* v1, long* v2) {
  char line[40], *ptr, *str;
  if (!readLine(file, line, sizeof(line))) {
    return false;  // EOF or too long
  }
  *v1 = strtol(line, &ptr, 10);
  if (ptr == line) return false;  // bad number if equal
  while (*ptr) {
    if (*ptr++ == ',') break;
  }
  *v2 = strtol(ptr, &str, 10);
  return str != ptr;  // true if number found
}

void setup(){
  long x, y;
  Serial.begin(9600);
  if (!SD.begin(SS)) {
    Serial.println("begin error");
    return;
  }
  file = SD.open("TEST.CSV", FILE_READ);
  if (!file) {
    Serial.println("open error");
    return;
  }
  while (readVals(&x, &y)) {
    Serial.print("x: ");
    Serial.println(x);
    Serial.print("y: ");
    Serial.println(y);
    Serial.println();
  }
  Serial.println("Done");
}
void loop() {}
 

Reading this "TEST.CSV" file.
Results in this output
Here is a simple sketch that reads a CSV file with two numbers on each line.  It doesn't use the dangerous String class.
Code: [Select]

#include <SD.h>
File file;

bool readLine(File &f, char* line, size_t maxLen) {
  for (size_t n = 0; n < maxLen; n++) {
    int c = f.read();
    if ( c < 0 && n == 0) return false;  // EOF
    if (c < 0 || c == '\n') {
      line[n] = 0;
      return true;
    }
    line[n] = c;
  }
  return false; // line too long
}

bool readVals(long* v1, long* v2) {
  char line[40], *ptr, *str;
  if (!readLine(file, line, sizeof(line))) {
    return false;  // EOF or too long
  }
  *v1 = strtol(line, &ptr, 10);
  if (ptr == line) return false;  // bad number if equal
  while (*ptr) {
    if (*ptr++ == ',') break;
  }
  *v2 = strtol(ptr, &str, 10);
  return str != ptr;  // true if number found
}

void setup(){
  long x, y;
  Serial.begin(9600);
  if (!SD.begin(SS)) {
    Serial.println("begin error");
    return;
  }
  file = SD.open("TEST.CSV", FILE_READ);
  if (!file) {
    Serial.println("open error");
    return;
  }
  while (readVals(&x, &y)) {
    Serial.print("x: ");
    Serial.println(x);
    Serial.print("y: ");
    Serial.println(y);
    Serial.println();
  }
  Serial.println("Done");
}
void loop() {}
 

Reading this "TEST.CSV" file.
Results in this output
@fat16lib, I am having a little difficulty in following your code; however with some time/google I think I will figure it out.  If you have a minute, could you look at my post and see if this code you posted would be applicable to my situation?  I have an SD card with the fixed-length HEX and then the username, can I use your code for this case?

https://forum.arduino.cc/index.php?topic=622216.0

Code: [Select]

ba322d387d980d74fc5d0d46, John
ba039b572f612e58d05a8c2d, Katie
979aeeb3ee35360a24d6190, Sandra
78cabc678dc0a503647cb05d, Tim



Title: Re: Read CSV or TXT from SD card into string or array
Post by: wildbill on Jun 20, 2019, 03:09 pm
To get started just use the readline function to pull each line in turn into a character array. Then print it to the serial port. Once that's working you can figure out how to parse it into two parts for storage.
Title: Re: Read CSV or TXT from SD card into string or array
Post by: voha888 on Sep 12, 2019, 11:32 pm
You can read SD card files line-by-line if this helps:

Code: [Select]
String l_line = "";
//open the file here
while (l_SDFile.available() != 0) {
    //A inconsistent line length may lead to heap memory fragmentation
    l_line = l_SDFile.readStringUntil('\n');
    if (l_line == "") //no blank lines are anticipated
      break;
    //

   //parse l_line here
}
//close the file here

  

Cheers

Catweazle NZ
Thank You! But some Arduino functions not work with strings created with
Code: [Select]
.readStringUntil('\n');
 function
Because files on SD have "CR" symbol, best way is using
Code: [Select]
.readStringUntil('\r');
 

//Sorry, for my bad English
Title: Re: Read CSV or TXT from SD card into string or array
Post by: sterretje on Sep 22, 2019, 05:03 pm
Because files on SD have "CR" symbol, best way is using
It depends on the OS and application that wrote the file to SD in the first place. See https://en.wikipedia.org/wiki/Newline#Representation (https://en.wikipedia.org/wiki/Newline#Representation).

PS
This thread is old