Wrong CSV values from SD card

Projects Programming
1

Hi, I'm trying to read a .csv file from a sd card. The format of the file is ID | Location | Color | Availability. It should be int | string | string | string. When using my code it reads the wrong values from the file. It also returns some squares for one of the strings.

#include <CSV_Parser.h>
#include <SPI.h>
#include <SD.h>

const int chipSelect = 10;
int test = 0;



void setup() {
  Serial.begin(9600);
  delay(3000); 

    if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    
    // don't do anything more:
    while (1);
  }
  Serial.println("card initialized.");

  CSV_Parser cp(/*format*/ "dsss", /*has_header*/ true, /*delimiter*/ ';');
  cp.parseLeftover();

  if (cp.readSDfile("/Land.csv")) {
    int16_t *ID = (int16_t*)cp["ID"];
    char    **Lokasjon = (char**)cp["Lokasjon"];
    char    **Farge = (char**)cp["Farge"];
    char    **Utvalg = (char**)cp["Utvalg"];


    Serial.print(ID[1],DEC); Serial.print(" - ");
    Serial.print(Lokasjon[1]); Serial.print(" - ");
    Serial.print(Farge[1]); Serial.print(" - ");
    Serial.print(Utvalg[1]); 


}

}

The output: 12 - Alaska - Lilla - �
NB : (The lastone is four squares!)
It should return: 2 - Alaska - Lilla - Rosa

What can cause this?

2

I don't know, but put the card into your PC and print some of the file in HEX and post on the thread.

3

I used cp.print(); and this prints the entire file in array format correctly. So the problems sits in my Serial.print function.

4

Indulge us.

Print the first few rows of your file in a manner that allows us to see what you feeding this.

a7

5

When plugged in to my pc and prited as HEX? Because I dont know how to do that.
The output from the IDE when using cp.print();:

5 | Albania | Rosa | AV
2 | Alaska | Lilla | Rosa
6 | Brazil | Bla | Lilla
1 | Danmark | Hvit | Bla
4 | Norge | Rod | Hvit
3 | Sverige | Gronn | Rod
7 | USA | Gul | Gronn
9 | Australia | Konge Bla | Gul
8 | Spania | Oransje | Konge Bla
10 | Portugal | AV | Oransje
6

Those lines would be trivial to parse using strtok().

7

Do I have to use strtok() when using CSV Parser?

8

How well is CSV Parser working for you?

1 Like
9

Haha good point. I just find it weird that it seems to read and parse my data correctly but I can't print it.

10

Does your file have headers?

Is the delimiter the semicolon ';' character?

What is the roll of the parseLeftover() method call where you make it?

a7

11

CSV files are Comma Separated Value, not "|" separated value.

Try this example using strtok():

void setup(){
  Serial.begin(115200);

char a[]="10 | Portugal | AV | Oransje";
char* token = strtok(a," |");
  while (token != NULL)
  {
    Serial.println(token);
    token = strtok (NULL, " |");
  }
}
void loop(){}

This is the result:

10
Portugal
AV
Oransje
12

Which arduino board are you using? The actual parsing and print statements are working, using a local char array instead of an SD card, but the code seems a bit memory intensive, so you may be short of ram.

14

I know. The file is seperated by ";". The cp.print(); just prints the file with the "|". I use ";" as delimiter.

15

So replace the ";" with comma! Why is this so hard?

The code posted in #11 will work with your file, if you replace "|" with ";".

16

A nano. It says in the IDE 56% programstorage space and 45% dynamic memory.

17

I suspect you need more ram. I modified your sketch to run from fixed data, and get correct output, but this will not run on an atmega328, uses too much memory (and I do not use the 512 bytes of ram used by the SD buffer):

#include <CSV_Parser.h>

void setup() {
  Serial.begin(9600);
  delay(3000);

  const char * csv_str = "ID;Lokasjon;Farge;Utvalg\n"
                         "5;Albania;Rosa;AV\n"
                         "2;Alaska;Lilla;Rosa\n"
                         "6;Brazil;Bla;Lilla\n"
                         "1;Danmark;Hvit;Bla\n"
                         "4;Norge;Rod;Hvit\n"
                         "3;Sverige;Gronn;Rod\n"
                         "7;USA;Gul;Gronn\n"
                         "9;Australia;Konge Bla;Gul\n"
                         "8;Spania;Oransje;Konge Bla\n"
                         "10;Portugal;AV;Oransje\n";

  Serial.println("Accessing values by column name:");
  CSV_Parser cp(csv_str,/*format*/ "dsss", /*has_header*/ true, /*delimiter*/ ';');
  //cp.parseLeftover();

  int16_t *ID = (int16_t*)cp["ID"];
  char    **Lokasjon = (char**)cp["Lokasjon"];
  char    **Farge = (char**)cp["Farge"];
  char    **Utvalg = (char**)cp["Utvalg"];


  Serial.print(ID[1], DEC); Serial.print(" - ");
  Serial.print(Lokasjon[1]); Serial.print(" - ");
  Serial.print(Farge[1]); Serial.print(" - ");
  Serial.print(Utvalg[1]);

  Serial.println();
  Serial.println();
  cp.print();
}

void loop() {

}

Output:

Accessing values by column name:
2 - Alaska - Lilla - Rosa

CSV_Parser content:
rows_count = 10, cols_count = 4
   Header:
      ID | Lokasjon | Farge | Utvalg
   Types:
      int16_t | char* | char* | char*
   Values:
      5 | Albania | Rosa | AV
      2 | Alaska | Lilla | Rosa
      6 | Brazil | Bla | Lilla
      1 | Danmark | Hvit | Bla
      4 | Norge | Rod | Hvit
      3 | Sverige | Gronn | Rod
      7 | USA | Gul | Gronn
      9 | Australia | Konge Bla | Gul
      8 | Spania | Oransje | Konge Bla
      10 | Portugal | AV | Oransje
Memory occupied by values themselves = 165
sizeof(CSV_Parser) = 44
18

This my be more efficient on memory and storage? This is just a sample file, the finished project would contain 370 rows. So I am actually looking for a method to just read on line from csv, do some stuff with it in the arduino, and the the next line and so on. Could this be better for that method?

19

Why does it show the low percentages even if it runs out of memory?

20

I ran the code on an UNO R4, the data is too large to even compile successfully on an UNO/Nano.

21

strtok() works in place, so it requires no new storage. It modifies the original input string in order to break it into C-string tokens (by replacing the delimiter characters with zero bytes).