Read data Column wise from CSV file stored in SD card and save as array variable

Using Arduino Programming Questions
1

Hi,

I need a help from the community to read the set of data points column wise and save complete 1st column's data set as a array variable in Arduino. Similarly, continue for n number of columns. The csv file must be read from SD card.

array a = [1985,1]
array b= [1985,2]
array c= [1985,3]
array d= [1985,4]
array e= [1985,5]

In the attached file, there are 5 columns each column consists of 1985 data points.

I am using Arduino Mega 2560 with external SD card reader

It will be appreciable if you either share the code or guide me to the right topic already posted related to this.

Thank you.

sample.txt (47.9 KB)

2 
array a = [1985,1]
array b= [1985,2]
array c= [1985,3]
array d= [1985,4]
array e= [1985,5]

What does the 1985 relate to and where did it come from ? It is not in your sample file

It looks like you would be better off using a 2 dimensional array for target
One row per data row and one column per data item

Read from the file and add each character to a temporary char array. When you get a comma add a trailing '\0' to the char[] variable to turn it into a c-string and use atoi() to convert it to an int variable and save it in the current column of the current row of the data array

data[currentRow][currentColumn] = atoi(charArray);

increment the currentColumn counter
Carry on reading characters, processing them, adding them to the data array until you get a carriage return signalling the end of row
Set currentColumn to zero and increment currentRow
Keep going until you have read the entire file

3

here's a function that works on my laptop to extract integers from a csv string that you may find helpful.

don't know if fopen works on an SD file. you need to store the integer using atoi() instead of just printing it.

void
read (
    char *filename)
{
    FILE  *fp;
    char   s[BUFSIZ];

    printf ("%s: %s\n", progname, filename);

    if ( (fp = fopen (filename, "rb")) == NULL)  {
        perror ("app - fopen input");
        exit (1);
    }

    while (fgets (s, BUFSIZ, fp) != NULL)  {
        s [strlen (s)-1] = 0;     // strip \n

        char *t = strtok (s, ",");
        while (t)  {
            printf (" %6d", atoi (t));
            t = strtok (NULL, ",");
        }
        printf ("\n");
    }
}
4

Looking at the sample file it has nearly 2000 rows in it so depending on which processor you are using then memory for the arrays may be a problem but I am not clear exactly what you want to do, so please give more details of the requirements and your hardware

5

Would be easier to have a multi-dimensional array or array of struct, but even with separate arrays it will be much faster to process all columns at the same time, reading the file from SD card is the slowest operation.

6

I was going to create an example of what I suggested in reply #1 but when I looked at the sample file and found that it had nearly 2000 rows, each of 5 integers I stopped because of memory constraints and posted reply #3 to clarify the requirement and the hardware being used

7

so maybe it's better to load values from the SD in chunks for processing or only read one column of data.

Jon Bentley (Programming Pearls) discussed algorithms for sorting large files on PDPs where the entire file could not fit in memory.

would be interesting to hear what the application is

8

Fairly easy using strtok() to locate the commas and atoi() to convert to integers. If you intend to process each column separately because of memory constraints that will necessitate reading the file multiple times, but presents no problem parsing each line, just ignore the unused column data.

9

david_2018:
Fairly easy using strtok() ...

see code i posted

10

Whilst this is all very interesting we still don't know what the OP really wants so I will keep a watch until he/she replies

11

Hi,

Apologies for replying delay. Purpose is to play the datalog.csv file in Arduino Mega 2560 which contains PWM signals of 4 servo actuators logged at 80 to 110 millisecond interval and 8 motors signals logged at 20 to 30 millisecond.

  1. Read the datalog.csv file from SD card and store it in array variable name as
    array Time delay of servo, array time delay of motors, Array motor1pwm..... Array motor8pwm, Array servo1pwm...... Array servo4pwm.

  2. Write the above Array variable (PWM signal) to PWM pins of Arduino Mega 2560 and execute it with the Time delay of servo and motors.

  3. So, this will run the 8 motors (controlled by ESC) and 4 servos connected to battery using datalog.csv file as reference.

Application: This setup is used to test the battery duration and life cycle for recorded data.

Please let me know if you need any further clarification from my end.

Thank you for responding to my question.

12

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.