What is the best way for Arduino to handle a large amount of continuous data fast?

I'm not sure if this is the best category to ask this in, or if this was the best way to phrase the question, but I'm working on a project and I could use input from people who are a bit more experienced than myself. I am very new to this, so I appreciate any help I can get!

Note: I'm using pyserial to communicate serially, tkinter to create the GUI, and of course, Arduino to contol my hardware.

The goal: Basically, the project needs to open an excel file using a GUI - the excel file holds data concerning relative pressures/forces that two large stepper motors (attached to screws to extend and contract) should apply to an object. The file has 3 columns: the time (each entry is separated by a small fraction of a second), the force for motor 1 at that time, and the force for motor 2 at that time.

I've been having a difficult time getting this to work. The machine has to run for hours at a time, and the time difference between each entry is very small, such that every time I've tried to send it continuously, I get no response from the Arduino.

Options:

  1. Would the best option be to try and send it and store it all at once? I know Arduino has very little space, but there is a way to increase the storage with SD cards and other devices - correct?
  2. Or would it be best to try and figure out how to send it very quickly and continuously? That seems to be the method that simplifies my code the most, but the Arduino is so unresponsive when that happens! Am I doing something wrong there?
  3. Another option I was considering would be to try and have the Arduino read directly from the excel file - which I have no idea if it is feasible or not.

I am very open to other ideas if anyone has any!

If it helps I also attached the code I have so far which hasn't been working. I'm trying the continuous stream method here and it's been rough.

Here is my relevant python code to send the data:

def sendValues():
    '''
    send a value over if the motor is on, then wait however long 
    '''
    global loadDiffValues
    global shearDiffValues
    global timeDiffValues
    global axialStatus
    global shearStatus
    while True:
        while axialStatus == "On" or shearStatus == "On":
            ser.reset_input_buffer()
            ser.reset_output_buffer()
            if axialStatus == "On":
                sendA = "A"+str(loadDiffValues[0])
                print('sent')
            else:
                sendA = 'a'
            ser.write(sendA.encode())
            ser.reset_input_buffer()
            ser.reset_output_buffer()
            if shearStatus == "On":
                sendS = "S"+str(shearDiffValues[0])
            else:
                sendS = 's'
            ser.write(sendS.encode())
            ser.reset_input_buffer()
            ser.reset_output_buffer()
            timeDiffValues[0] = float(timeDiffValues[0]) 
            sendT = "T"+str(timeDiffValues[0])
            ser.write(sendT.encode())
            ser.reset_input_buffer()
            ser.reset_output_buffer()
            time.sleep(float(timeDiffValues[0])) //this is around 0.01 or so
            loadDiffValues = loadDiffValues[1:]
            shearDiffValues = shearDiffValues[1:]
            timeDiffValues = timeDiffValues[1:]

and here is the corresponding Arduino code:

boolean axialStatus = 0;  //start with both motors off
boolean shearStatus = 0;

double pulse_delay(double posChange, double transitionTime);

float time_value;
float axial_value;
float shear_value;

void setup() {
  pinMode(PulseAxialV, OUTPUT); pinMode(DirAxialV, OUTPUT); pinMode(PulseShearV, OUTPUT); pinMode(DirShearV, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  unsigned long currentTime = millis();
  if (Serial.available()>0){
    String value = Serial.readString();
    if (value.substring(0,1)=="A"){
      previousTimeAxial = millis();
      axialStatus = 1;
      float axial_value = value.substring(1).toFloat();
      if (axial_value < 0){
        digitalWrite(DirAxialV, LOW);
      }
      else if (axial_value > 0){
        digitalWrite(DirAxialV, HIGH);
      }
    }
    else if (value.substring(0,1)=="S"){
      previousTimeShear = millis();
      shearStatus = 1;
      float shear_value = value.substring(1).toFloat();
      if (shear_value < 0){
        digitalWrite(DirShearV, LOW);
      }
      else if (shear_value > 0){
        digitalWrite(DirShearV, HIGH);
      }
    }
    else if (value.substring(0,1)=="T"){
      float time_value = value.substring(1).toFloat();
    }
    else if (value.substring(0,1)=="a"){
      axialStatus = 0;
    }
    else if (value.substring(0,1)=='s'){
      shearStatus = 0;
    }
    else if (value.substring(0,1)=='l'){
      axialStatus = 0;
      shearStatus = 0;
    }
  }
  if (axialStatus == 1){
    if (currentTime - previousTimeAxial < pulse_delay(axial_value, time_value)){
      digitalWrite(PulseAxialV, HIGH);
    }
    if (currentTime - previousTimeAxial < 2*pulse_delay(axial_value, time_value)){
      digitalWrite(PulseAxialV, LOW);
    }
    else {
      previousTimeAxial = currentTime;
    }
  }
  else {
    digitalWrite(PulseAxialV, LOW);
  }
  if (shearStatus == 1){
    if (currentTime - previousTimeShear < pulse_delay(shear_value, time_value)){
      digitalWrite(PulseShearV, HIGH);
    }
    else if (currentTime - previousTimeShear < 2*pulse_delay(shear_value, time_value)){
      digitalWrite(PulseShearV, LOW);
    }
    else {
      previousTimeShear = currentTime;
    }
  }
  else {
    digitalWrite(PulseShearV, LOW);
  }
}


//the rest isn't too important for my problem, ignore it
float coeff = 0.45714;

double pulse_delay(double posChange, double transitionTime){
  if (posChange == 0){
    return 0.0;
  }
  else{
    double pd = coeff*transitionTime*1250/posChange;
    return pd;
  }
}

Please! A large amount of continuous data fast conveys NO usable information. If you put some REAL numbers to those and explained what the Arduino is supposed to do with that, then we might be able to offer some solutions.

Hi, @megby
Welcome to the forum.
Please read the post at the start of any forum , entitled "How to use this Forum".

This will help with advice on how to present your code and problems.

What exactly is your project?

Thanks... Tom... :smiley: :+1: :coffee: :australia:

time.sleep(float(timeDiffValues[0])) //this is around 0.01 or so

It looks like you send data every 10ms. The packets and their size are unclear to me.

I would say the first thing to do is to speed up the serial interface. Is there some reason for the low baud rate?

Serial.begin(9600);

Your code assumes there is always a single fully received message in the Arduino receive buffer. In practice you may also have a partly received message, or even multiple messages concatenated head to tail.

Firstly you need to add 'message framing' at the sending end so the Arduino receiver can distinguish one message from the next. For example you send "message" as "<message>"

For the receiver, one solution is to extract data using Serial.readStringUntil('>') and then check that the first character is a '<' to verify that you have a complete message, not just a fragment. Note: readStringUntil is a blocking function so you need to set an appropriate timeout value that is compatible with the rest of your code.