how to read long csv and save it value to matrix ?

my question ( explanation below ):
#1. How do i effectively read long csv using sdfatlib
#2. is there any solution to stdio.h for feof ( read whole file ), fscanf ( change reading data to integer for me)

here’s my pseudocode :
#1. know how much integer in csv
#2. using malloc to provide 16-bit array with csv integer length
#3. get every integer from csv to array
#end return array
below i add csv that i want to read in my arduino uno ( lena.csv )? , any guidance here ?

i tried to modified readCSV example from sdfatlib http://code.google.com/p/sdfatlib/

this is what i changed :

void readFile() {
  //int nilai;
  long lg;
  float f1, f2;
  char c1, c2; // ;
  int jumlahPix=0;
  
  // open input file
  ifstream sdin(fileName);
  
  // check for open error
  if (!sdin.is_open()) error("open");
  
  // read until input fails
  // while (sdin >> lg >> c1 >> f1 >> c2 >> f2) {
    while (sdin >> f1 >> c1 ) {
    
    // error in line if not commas
    if (c1 != ',' ) error("comma"); // || c2 != ',') error("comma");
    
    // print in six character wide columns
    cout << setw(6) << f1 << endl;
    
    jumlahPix++;
  }
  char c3 = 10;
  while (sdin >> f1){ 
  // print in six character wide columns
    cout << setw(6) << f1 << endl;
    
    jumlahPix++;
  }
  // Error in an input line if file is not at EOF.
  if (!sdin.eof()) error("readFile");
  // Tampilkan jumlah pixel 
  cout << "jumlah pixel : " << jumlahPix << endl;
}
//------------------------------------------------------------------------------
// write test file
void writeFile() {

  // create or open and truncate output file
  ofstream sdout(fileName);
  
  // write file from string stored in flash
  sdout << pstr(
    "1,2.3,4.5,"
    "6,7.8,9.0,"
    "9,8.7,6.5,"
    "-4,-3.2,-1\n") << flush;

  // check for any errors
  if (!sdout) error("writeFile");
  
  // file is closed by destructor when it goes out of scope.
}

it is what the result :

1.00
2.30
4.50
6.00
7.80
9.00
9.00
8.70
6.50
-4.00
-3.20
jumlah pixel : 11 // the result in this line should be 12
Done

if the format look like the example :

%f,%f,%f

← it is what i mean complete fix format

so the problem :

%f,

, continously till value of -1, but it doesn’t get proper answer cause of none fixed format

what i thought is when we provide the complete fix format is easy for short length csv ?
but how to apply it in long length csv ? :frowning:

and heres

FILE *fp = fopen(filename, "rb");
  if (fp == NULL) {
    fprintf(stderr, "Can't open input file\n");
    exit(1);
  }
  int value;
  //first estimate no of pixels:
  int NoPix=0;
  while (!feof(fp)) {fscanf(fp,"%i ",&value);NoPix++;}

what should be like in c programming language, in the last line it skip white space and save the value as integer

and i had search another solution, like Input/output via <iostream> and <cstdio>, C++ FAQ

in the section [15.12] How can I open a stream in binary mode?, but when i done i get char which cannot be processed as integer

and the last one, i read avr-libc: <stdio.h>: Standard IO facilities
but it didn’t clearly understand by me

lena.csv (235 KB)

lena2.csv (253 KB)

the whole trick with parsing CSV files is 1) read one line ==> that is read until line delimiter like \n or \r\n 2) break the line in fields using the field delimiter 3) assign the fields to the appropiate variable (optional convert them e.g. string to int, ENUM to byte whatever) 4) process the line (mostly csv files are processed line by line) 5) goto 1 ...

(2) can have some trouble as you may have - a field enclosed in "" - escaped characters like \" \, \; and of course \

Above is the easiest way IF you have enough RAM to hold a line


second way is as follows

Have a fieldcounter =0; 1) read field ==> read until line delimiter OR field delimiter // (2) remarks apply here too 2) assign to appropiate variable 3) if field was last of the line => process the line (mostly csv files are processed line by line) AND reset the field counter 4) goto 1 ...

Not so different, but it needs some extra care tokeep the field count correct.... But it uses a lot less RAM as you only need to hold one field + one separater at max.

Succes!

what i confused here is " how do i use delimiter inside ?"

 while (sdin.getline(buffer, line_buffer_size, '\n') || sdin.gcount())

here as i know, it read every line

and if i use modify the code

while (sdin.getline(buffer, line_buffer_size, '\n') || sdin.gcount()) { 
        while (sdin.getline(buffer, line_buffer_size, '\,') || sdin.gcount()){ 
       ... code here }
}// end of line

(1) from above code : first line loop till the ‘\n’ but i don’t know what object to hold that string for limiter ?
( 2 ) do i have to write in another csv and read it ?

( 3 ) or please have a code example ?

is it look like this ?
what i confused is what object do i have to hold the buffer ? for get the variable

You can read a huge file, only storing a handful of bytes, if you use a state machine. I discuss this here:

http://www.gammon.com.au/serial

How does the expression go? How do you eat an elephant? One bite at a time.