High speed data transfer to either PC or SD-card

Iam working on a project where iam reading out a matrix of 32x32 in total 1024 measurments each measurments is between 0 and 4000. eventually i want to get a framrate of +-50Hz on a matrix with +-1500 points. but to reach that i need a way to store data quickly or move data to my PC quickly and store it there. I want to save these measurments on either a sd card connect to my esp32 devkit or send it to my pc. Right now iam sending the data to my pc over usb using serial.print.
My current code measures 1024 data points in 80ms.I know a bottleneck of the speed is the adc but i will replace this with an external one. How fast would data storage to an sd-card be? and could i get faster speeds transfering over wifi or bluetooth? In short i need to save 1500 points wich value ranges from 0to4000 at a rate of +-50Hz. Iam open to other microcontrollers.

this is my current code

Any advice will be appreciated

#include <Shifty.h>

Shifty myreg;

// Pin definitions shift register
int latchPin = 22;   // Latch pin of 74HC595 (RCLK)
int clockPin = 18;   // Clock pin of 74HC595 (SRCLK)
int dataPin = 4;     // Data pin of 74HC595 (SER)
int ResetHigh = 21;  // Master Reset pin, active LOW (set HIGH to enable)
int UseLow = 19;     // Use Enable pin, active HIGH (set LOW to enable)
int Power = 23;      // Gives power to shift registers
int Order[] = {24, 25, 26, 27, 28, 29, 30, 31, 16, 17, 18, 19, 20, 21, 22, 23, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7};

// Pin definitions Multiplexer
int CS_in = 25;
int WR_in = 33;
int EN = 32;
int S_in[5] = {17, 13, 14, 27, 26};
const int INPIN = 34;

void setup() {
  Serial.begin(921600); // Start Serial communication
  
  // Set pin modes for shift register
  pinMode(latchPin, OUTPUT);
  pinMode(clockPin, OUTPUT);
  pinMode(dataPin, OUTPUT);
  pinMode(ResetHigh, OUTPUT);
  pinMode(UseLow, OUTPUT);
  pinMode(Power, OUTPUT);

  // Initialize shift register
  digitalWrite(ResetHigh, HIGH); // Keep the shift register active
  digitalWrite(UseLow, LOW);     // Enable outputs
  digitalWrite(Power, HIGH);     // Power the shift registers
  digitalWrite(ResetHigh, LOW);
  digitalWrite(ResetHigh, HIGH);

  // Input for Shifty library
  myreg.setBitCount(32);
  myreg.setPins(dataPin, clockPin, latchPin);

  // Reset all outputs to 0
  clearShiftRegister();

  // Set pin modes for multiplexer
  pinMode(CS_in, OUTPUT);
  pinMode(WR_in, OUTPUT);
  pinMode(EN, OUTPUT);

  pinMode(INPIN, INPUT);

  // Initialize multiplexer
  digitalWrite(CS_in, LOW);
  digitalWrite(WR_in, LOW);
  digitalWrite(EN, LOW);

  for (int x = 0; x < 5; x++) {
    pinMode(S_in[x], OUTPUT);
    digitalWrite(S_in[x], LOW);
  }
}

void loop() {
  char buffer[50]; // Buffer for formatted data
  unsigned long startTime = millis();

  for (int i = 0; i < 32; i++) { 
    myreg.writeBit(Order[i], HIGH);

    for (int i_in = 0; i_in < 32; i_in++) {
      if (i_in <= 15) {
        pinSelect_in(i_in);
      } else {
        pinSelect_in(47 - i_in);
      }

      int sensorValue = analogRead(INPIN);
      sprintf(buffer, "%d,%d\n", i * 32 + i_in, sensorValue);
      Serial.print(buffer); // Send formatted data in a single call
    }           
    myreg.writeBit(Order[i], LOW);
  }

  unsigned long endTime = millis();
  sprintf(buffer, "execution was %lu\nstart %lu\nend %lu\n", 
          endTime - startTime, startTime, endTime);
  Serial.print(buffer);

}

// Function to clear the shift register (turn all outputs OFF)
void clearShiftRegister() {
  digitalWrite(latchPin, LOW); // Prepare to send data
  shiftOut(dataPin, clockPin, LSBFIRST, 0b00000000); // Send all 0s
  digitalWrite(latchPin, HIGH); // Latch the data
}

void pinSelect_in(int pinnum_in) {
  digitalWrite(WR_in, LOW);
  for (int x = 0; x < 5; x++) {
    byte state = bitRead(pinnum_in, x);
    digitalWrite(S_in[x], state);
  }
  digitalWrite(WR_in, HIGH);
}

50 x 1500 points = 75000 points x 2 bytes = 150000 bytes = 1.2 Mbs

Compression / packing is possibly part of an answer.
The values fit in 12 bit so 6 bytes could hold 4 values. Saves 25% IO.

Do the values change a lot between samples?
if not delta coding could help.

Storage to an SD card is ALWAYS in 512 byte blocks. The time to write that block and do other necessary SD card functions is always blocking your Arduino program until the write is complete.
Best to send the data to a PC where you use the power of the operating system to save the data in a non-blocking manner.

1 Like

yes the values change a lot, but i have never heard of delta coding so i will still give it a look thank you!

thanks for the info on the sd-card, by wire i only get 80ms per transfer tho. i tried wifi but it became a lot worse at around 600ms would bluetooth be faster? or am i doing something really wrong with the wifi?

I'm thinking ethernet cable to PC.

Delta coding is a generic term for recoding data based upon differences.
This can have several forms (mention 3 to get the idea)

  • you have a reference frame and you save the deltas compared to that.
    These deltas might be all 8 bit instead of 16 bit.
  • you store only the cells that changed, you must add position {row, column, new value}, this can be e.g. row optimized.
  • you encode every cell compared to the previous value (e.g. same row or same column), this assumes that cells that are side by side do not differ much.

Note, how well compression works depends on the data and the changes, This can save a lot of bytes when there are only a few changes however it can also blow up the size of the data when there are a lot of changes.