CSV to array with strtok()

Projects Programming
1

I want to put rows from a CSV file into an array. The CSV is full of milliseconds and looks like this.

1111,2222,3333
4444,5555,6666
7777,8888,9999
0000,1111,2222
3333,4444,5555

So far I can read from the CSV with a whole row at a time and get this in the serial monitor.

111122223333
444455556666
777788889999
000011112222
333344445555

My code is this.

#include <SPI.h>
#include <SdFat.h>
const int SD_CS = 8;
SdFat SD;
File logFile;


void setup() 
{
  Serial.begin(9600);
  pinMode (SD_CS, OUTPUT);
  while(!SD.begin(SD_CS)); 
  csvToArray();
}


void loop() 
{
}


void csvToArray()
{
  static String csvRow = "";

  logFile = SD.open("test_SdA.csv");

  while (logFile.available())
  {
    int csvChar = logFile.read();
    if (isDigit(csvChar)) csvRow = csvRow + (char)csvChar;

    if (csvChar == '\\n')
    {
      Serial.println(csvRow.toInt());
      csvRow = "";
    }
  }
  Serial.println("---- End of CSV");
  logFile.close();
}

I want to split the CSV with something like this so I can put the values into an array.

csvCell1 = strtok(csvRow, ",");
csvCell2 = strtok(NULL, ",");
csvCell3 = strtok(NULL, ",");

I've tried to define csvCell1 as a String or a char but the sketch doesn't compile because it's the wrong variable type.
What am I getting wrong?

2

Will fix your compile error.

Not that you'll be able to use strtok on a String. That's for char arrays.

Off the top of my head:

char buf[80];
char *csvCell1;
csvRow.toCharArray(buf, sizeof buf);
csvCell1 = strtok(buf, ",");

etc.

3

The problem is that strtok() needs a character array. It cannot directly process Arduino String objects. Your code also removes commas while reading. Therefore, strtok() has no separators available. Keep the commas when constructing each row. Then, copy the row into a char[] buffer. You can use strtok() to extract each field. However, parsing numbers directly is better for Arduino. Build each number digit-by-digit while reading. Store completed values directly into a two-dimensional array. This approach uses less memory and avoids String fragmentation. It is especially suitable for SD card CSV processing.

4

if you read a line of text, e.g. "444,5555,6666", into a char array you could use sscanf() to parse the data into an array, e.g.

sscanf(text, "%d,%d,%d", &d[0], &d[1], &d[2])

sample program reading from serial monitor using an ESP32

void setup() {
  Serial.begin(115200);
  delay(2000);
  Serial.println("enter three integers comma seperated");
  while (1) {
    char text[100] = { 0 };
    //  read line of text terminated by \n into array text[100]
    if (Serial.readBytesUntil('\n', text, 100) == 0) continue;
    // text entered print it
    Serial.printf("\ntext %s\n", text);
    // use sscanf() to parse three integers into array d[3]
    int d[3] = { 0 };
    if (sscanf(text, "%d,%d,%d", &d[0], &d[1], &d[2]) == 3)
      // scanf() converted three integers OK print them
      for (int i = 0; i < 3; i++)
        Serial.printf("d[%d]=%d ", i, d[i]);
    else  // scanf() failed
      Serial.print("scanf failed");
  }
}

void loop() {}

sample output when strings "1,2,3" and "4444,5555,6666" entered

enter three integers comma seperated

text 1,2,3
d[0]=1 d[1]=2 d[2]=3 
text 4444,5555,6666
d[0]=4444 d[1]=5555 d[2]=6666
5

Thanks everyone. I undertand I need t to change csvRow to a character array, not a String, but unfortunately I'm not very good at this yet and didn't understand very much of else that you've said.

6

There are plenty of tutorials on line explaining how to use strtok and character arrays, also called "C-strings".

Search terms like "strtok tutorials" work well.

7

Yeah, I know. And I've literally been trying to work this out for 12hrs with all the tutorials I can find and I still haven't been able to get it to work. Telling me to to use Google isn't very helpful.

8

Post #2 showed you what to change. I didn't tell you Google anything. If you have a question about something you didn't understand, then ask it. Just throwing your hands in the air is not helpful.

9

@van_der_decken Sorry. That was in reply to @jremington, telling me to search for stuff and it wasn't to you.
Thank you for your help. I'm really struggling here.

You said if (csvChar == '\n') will fix the compile error. Isn't that what I've already got? Or do you mean I need to change it? And if so, what do I need to change it to?

Thank you for the code you posted but I don't understand what it's doing or how to use it.

char buf[80];
char *csvCell1;
csvRow.toCharArray(buf, sizeof buf);
csvCell1 = strtok(buf, ",");

Ok, so I create an array called buf that's 80 characters(?) long.
And I think a "pointer", but I'm getting out of my depth already.
If I change declaring csvRow to a char array, I don't know what .toCharArray does, though I understand that sizeof buf says how many characters (or maybe bytes?) have been stored in buf.
And I think I understand csvCell1 = strtok(buf, ","); does. I assume I need to declare char csvCell1 before that. Though I'm not sure why char csvCell1 = strtok(buf, ","); wouldn't work.

10

Are they always in threes? Are they always separated by commas? What's the line ending char?

Enquiring minds need to know!

11

At the moment, yes, there are always three values in each row of the CSV, they're always separated by commas (I thought that was kind of the definition of Comma Separated Value / CSV files?) and I'm expecting the line to end with \n, but I could be wrong about that.

I'm kinda new to this so I don't know what is useful to tell you. :rofl:

12

I think I understand what you're talking about, but how do you build each number digit by digit?

13
  1. Declare an int variable named total and a char variable named newChar
  2. Set total to zero
  3. Read a character and save it to newChar
  4. If newChar is a comma then the total variable holds your number ready to save to your array
  5. Else multiply total by 10 and add the digit represented by newChar to it
  6. Keep going until you read a comma in step 3

In step 5 you can convert the ASCII value of newChar into a digit by subtracting 48 from it

14

As mentioned, sscanf is safer and easier than strtok. And while String can cause problems, it does have its conveniences. If you already have one, there's no need to copy into yet another char buffer; it already has one.

void setup() {
  Serial.begin(115200);
  String line{ "1234,5678,9012" };
  int d[3];
  int matched = sscanf(line.c_str(), "%d,%d,%d", &d[0], &d[1], &d[2]);
  if (matched == 3) {
    Serial.print(d[0]);
    Serial.print('\t');
    Serial.print(d[1]);
    Serial.print('\t');
    Serial.println(d[2]);
  } else {
    Serial.print("didn't work; matched: ");
    Serial.println(matched);
  }
}

void loop() {}
15

input

1111,2222,3333
4444,5555,6666
7777,8888,9999
0000,1111,2222
3333,4444,5555

output

    {   1111,   2222,   3333, },
    {   4444,   5555,   6666, },
    {   7777,   8888,   9999, },
    {      0,   1111,   2222, },
    {   3333,   4444,   5555, },

laptop code

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_TOKS   10
char * toks [MAX_TOKS];
int    vals [MAX_TOKS];
int    nTok;

// -------------------------------------
void
tokDisp ()
{
    printf ("  tokDisp:\n");
    for (int n = 0; n < nTok; n++)
        printf (" %6d %s\n", vals [n], toks [n]);
}

// -------------------------------------
int
tokenize (
    char *s )
{
    char buf [90];

    nTok = 0;
    strcpy (buf, s);

    toks [nTok] = strtok (buf, ",");
    vals [nTok] = atoi (toks [nTok]);

    for (nTok++; (toks [nTok] = strtok (NULL, ",")); nTok++)
        vals [nTok]   = atoi (toks [nTok]);

    return nTok;
}

// -----------------------------------------------------------------------------
int
main (
    int    argc,
    char **argv )
{
    FILE *fp;

    if (argc < 2)  {
        fprintf (stderr, " need filename");
        exit (1);
    }

    fp = fopen (*++argv, "r");
    if (NULL == fp)  {
        fprintf (stderr, " invalid filename");
        exit (1);
    }

    char buf [90];
    while (NULL != fgets (buf, sizeof(buf)-1, fp))  {
        buf [strlen (buf)-1] = '\0';      // strip linefeed
     // printf ("  %s\n", buf);

        int nTok = tokenize (buf);

        printf ("    {");
        for (int n = 0; n < nTok; n++)
            printf (" %6d,", vals [n]);
        printf (" },\n");
    }

    return 0;
}
16

Here, you can try it like this..

Oh wait. Read it from a file.. Missed that bit. I thought you were going to copy paste the data into the serial monitor.

17

You made such basic, fundamental errors, (e.g. use of String objects), that I assumed you had not consulted any tutorials. If you have, then the problem appears to be that you simply aren't paying attention, and need instruction on the basics.

Hence, my mention of character arrays (aka C-strings), which is the starting point for use of strtok().

18

Wait, he wants it to convert to source code? Not just read them?

19

May be something like this (typed here from my iPhone - untested)

#include <SPI.h>
#include <SdFat.h>

constexpr uint8_t SD_CS = 8;
SdFat SD;
File logFile;

constexpr uint16_t maxNumberOfLines = 50;
constexpr uint16_t numberOfRecordsPerLine = 3;
constexpr uint16_t maxDigitsPerValue = 10;
constexpr uint16_t lineBufferSize = numberOfRecordsPerLine * (maxDigitsPerValue + 1) + 1;

struct CsvLine {
  uint32_t values[numberOfRecordsPerLine];
};

CsvLine data[maxNumberOfLines]; // one global buffer 
uint16_t lineCount = 0;

void readFile(const char * filename) {
  logFile = SD.open(filename);
  if (!logFile) {
    Serial.println("Could not open file");
    return;
  }

  char lineBuffer[lineBufferSize];
  lineCount = 0;

  while (logFile.available() && lineCount < maxNumberOfLines)  {
    size_t len = logFile.readBytesUntil('\n', lineBuffer, sizeof lineBuffer - 1);
    lineBuffer[len] = '\0'; // make sure it’s a cString

    char * token = strtok(lineBuffer, ",");
    for (uint16_t i = 0; i < numberOfRecordsPerLine && token != nullptr; i++)
    {
      data[lineCount].values[i] = strtoul(token, nullptr, 10);
      token = strtok(nullptr, ",");
    }

    lineCount++;
  }

  logFile.close();
}

void dumpData() {
  for (uint16_t i = 0; i < lineCount; i++) {
    for (uint16_t j = 0; j < numberOfRecordsPerLine; j++) {
      Serial.print(data[i].values[j]);
      if (j < numberOfRecordsPerLine - 1) Serial.write(",");
    }
    Serial.println();
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(SD_CS, OUTPUT);
  while (!SD.begin(SD_CS)) yield();

  readFile("test_SdA.csv");
  dumpData();
}

void loop() {}

Use 115200 for the serial monitor speed and note the code has no error checking, it expects a correctly formatted file

20 
#include <SPI.h>
#include <SdFat.h>
const int SD_CS = 8;
SdFat SD;
File logFile;

int value[100][3];

void setup() 
{
  Serial.begin(9600);
  pinMode (SD_CS, OUTPUT);
  while(!SD.begin(SD_CS)); 
  csvToArray();
}


void loop() 
{
}


void csvToArray()
{
  int r = 0;
  int c = 0;
  int v = 0;

  logFile = SD.open("test_SdA.csv");

  while (logFile.available())
  {
    char csvChar = logFile.read();

    if (isDigit(csvChar))
    {
      v = 10 * v + csvChar - '0';
    }
    else if (csvChar == ',')
    {
      Serial.print(v);
      Serial.print(" ");
      value[r][c] = v;
      c++;
      v = 0;
    }
    else if (csvChar == '\n')
    {
      Serial.println(v);
      value[r][c] = v;
      r++;
      c = 0;
      v = 0;
    }
  }
  Serial.println("---- End of CSV");
  logFile.close();
}

Edited to fix errors spotted by @alto777 @J-M-L