Help importing data from very large CSV

Hi, I need help efficiently reading in each row of a CSV when it is needed (all 12 columns) and saving it in an array called positions (as ‘long’ variable type, because I am saving large numbers with many decimal places and using +/- signs). I am going to be using matlab to generate some very complex motion profiles and I need to make a simple controller to follow those with a 3D printer. I am using a mega.

I’ve looked through a lot of forms and the sample code I’ve been able to find does not work the way i need it to, and I do not understand it enough to modify it to work. Here is the code structure template:

#include <AccelStepper.h>
#include <MultiStepper.h>
#include <SPI.h>
#include <SD.h>

File myFile;
const int chipSelect = 53;
long positions[12]; // Array for controlling steppers (I know that this library can only handle 10 stepper 
 //motors, I only have 4 stepper motors, but I have X_position, X_velocity, X_accel.... saved in the CSV)

AccelStepper stepper1(AccelStepper::FULL2WIRE, 3,4);
AccelStepper stepper2(AccelStepper::FULL2WIRE, 5,6);
AccelStepper stepper3(AccelStepper::FULL2WIRE, 7,8);

MultiStepper steppers;



void setup() {

  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
if (!SD.begin(chipSelect)) {
    Serial.println("initialization failed!");
    return;
  }

          
  myFile = SD.open("book1.csv",FILE_READ);
                      // here i need to initialize the first row and all 12 of the columns of the CSV and save them to a matrix/arrray "positions"
                      // the numbers are for example 3045.5467 and they are signed too
                      // i was thinking of then storing the "position()" in the CSV we read the last number
                      // then close the CSV to save memory
  positions[0] = 15000; // example X
  positions[1] = 15000; // example Y
  positions[2]=  15000; // example Z                      

  
// Configure each stepper
  stepper1.setMaxSpeed(10000);
  stepper2.setMaxSpeed(10000);
  stepper3.setMaxSpeed(10000);
  
  stepper1.setAcceleration(1000);
  stepper2.setAcceleration(1000); 
  stepper3.setAcceleration(1000);
  
// Then give them to MultiStepper to manage
  steppers.addStepper(stepper1);
  steppers.addStepper(stepper2);
  steppers.addStepper(stepper3);
}

void loop() {
  
   
  steppers.moveTo(positions);
  


        // while the stepper motors are moving I need to read in the next row of the CSV to replace positions[]... really need a function call here to read in the data to make the code cleaner
myFile = SD.open("book1.csv",FILE_READ);
                      //maybe at this point we can use the seek() based on the "position()" of the last read 
                      //here i need to read in row 2 and all 12 of the columns of the CSV and save them to the matrix/arrray "positions"
                      // the numbers are for example 3045.5467 and they are signed too
                      // i was thinking of then storing the "position()" within the CSV we read the last number
                      // then close the CSV to save memory

  //positions[0] = X_position is from CSV_column_1;
  //positions[1] = Y_position is from CSV_column_2;
  //positions[2]=  Z_position is from CSV_column_3;
  //positions[4] = X_velocity is from CSV_column_4;
  //positions[7] = X_acceleration is from CSV_column_7; // and so on through Y, Z and the extruder
  //positions[11] = CSV_column_12; 



  steppers.runSpeedToPosition(); // Blocks until all are in position

  

}

Further information/thoughts:
I can change the format of the CVS or change to a TXT or similarly common file type if it makes it easier. I also could actually make multiple CSV files since opening a single large file might use up all of my memory. I’m going to use Matlab to read in the GCODE and do a bunch of “fancy” controls computation for trajectory generation. Then I’m going to produce a file with every X,Y,Z: position, velocity, and acceleration setting for the entire print.

I can and have read/written to a TXT and CSV file with the arduino, so at least I know the wiring is right, but I need help with the software.

Lastly, does the “stepper1.setAcceleration(1000);” also set the deceleration limit?

Once I get this working I think I can produce some pretty nice improvements for 3D printer trajectory generation, but as a mechanical engineer my CS skills are lacking. Hopefully I’ll be able to work with this community or the Marlin community to convert what improvements I can provide with matlab code to the opensource arduino portfolio. This is a semester project for me which has limited time, but my professor PhD student was able to increase the average printer’s speed 2X without loosing any quality, (but that was ran a super powerful computer controller).

Much appreciated

The file size on an SD card does not matter and .csv (comma separated text value) files are easy to read.

What matters is that you have very little RAM memory to store data.

Ok, i thought using the "SD.open" loads the entire CSV into memory, but if not then I just need to overwrite the old stored values each time i read the code.

But how do I read in the columns and store them as 'long' variable type? and then the next time i read from the CSV make sure I've incremented to the next row?

There are various tutorials on line describing how to read text data from a file. You might also find Serial Input Basics helpful.

harderic:
Ok, i thought using the "SD.open" loads the entire CSV into memory, but if not then I just need to overwrite the old stored values each time i read the code.

Think about it, the Uno has 2K of RAM and SDcards can be 32GB... 7 orders of magnitude difference

But how do I read in the columns and store them as 'long' variable type? and then the next time i read from the CSV make sure I've incremented to the next row?

By parsing on the fly. Parsing is reading a string of characters and breaking it up into the structure you want,
and here you probably need simple ad-hoc parsing using a small state-machine. Without programming
experience you will be on a learning curve, but a worthwhile one(!)