Efficient way to save data from arduino sensors

Hi community!
I have analog accelerometer that I connect to arduino and can read the data and print in Serial. I connect to arduino from PC with USB and can save the outputs from Serial. The frequency of my accelerometer can be 500 Hz and 5 kHz that is important to note. I want to have opportunity to save the data as at max frequency as it possible.
There is the sketch:

// Make sure these two variables are correct for your setup
int scale = 200; // 3 (3g) for ADXL337, 200 (200g) for ADXL377
boolean micro_is_5V = true; // Set to true if using a 5V microcontroller such as the Arduino Uno, false if using a 3.3V microcontroller, this affects the interpretation of the sensor data

void setup()
{
  // Initialize serial communication at 115200 baud
  Serial.begin(115200); 
}

// Read, scale, and print accelerometer data
void loop()
{
  
  // Get raw accelerometer data for each axis
  int rawX = analogRead(A0);
  int rawY = analogRead(A1);
  int rawZ = analogRead(A2);

  
  // Scale accelerometer ADC readings into common units
  // Scale map depends on if using a 5V or 3.3V microcontroller
  float scaledX, scaledY, scaledZ; // Scaled values for each axis
  if (micro_is_5V) // Microcontroller runs off 5V
  {
   scaledX = mapf(rawX, 0, 1023, -scale, scale); // 3.3/5 * 1023 =~ 675
   scaledY = mapf(rawY, 0, 1023, -scale, scale);
   scaledZ = mapf(rawZ, 0, 1023, -scale, scale);

  }
  else // Microcontroller runs off 3.3V
  {
    scaledX = mapf(rawX, 0, 675, -scale, scale);
    scaledY = mapf(rawY, 0, 675, -scale, scale);
    scaledZ = mapf(rawZ, 0, 675, -scale, scale);
  }
  
  Serial.print(scaledX); Serial.print(";");
  Serial.print(scaledY); Serial.print(";");
  Serial.print(scaledZ); Serial.print(";");

}

// Same functionality as Arduino's standard map function, except using floats
float mapf(float x, float in_min, float in_max, float out_min, float out_max)
{
  return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}

As output I have the next (acceleration in g in x, y and z direction respectively) :

3.52;0.54;-1.25		
9.15;-0.53;-2.89		
9.20;-0.54;-2.91		
9.23;-0.53;-2.88	

The problem is that accelerometer with arduino should be located far from PC or notebook approx 30-35 meters. So I decided to write the data on SD Card. I found the example from source code of SD library (SD/Datalogger.ino at master ยท arduino-libraries/SD ยท GitHub) and measured the time of opening, writing and closing the file like that:

void saveSD()
{
  // create file for saving data
  start_open_t=micros(); // time in milli seconds
  File dataFile = SD.open("datalog.txt", FILE_WRITE);
  delta_open_t=micros() - start_open_t ;
  // if file is available for writing 
  if (dataFile) {
    // save data
    start_write_t=micros(); // time in milli seconds
    dataFile.println(dataString);
    delta_write_t=micros() - start_write_t ;
    // close the file
    start_close_t=micros(); // time in milli seconds
    dataFile.close();
    delta_close_t=micros() - start_close_t ;
    // print the messages
    Serial.println("Save OK");
    Serial.println("Open time:");
    Serial.println(delta_open_t);
    Serial.println("Write time:");
    Serial.println(delta_write_t);
    Serial.println("Close time:");
    Serial.println(delta_close_t);
  } else {
    // if file is not available
    Serial.println("Error opening datalog.txt");
  }
}

Measured time:

Open time:
7656
Write time:
2176
Close time:
8076

The open time is increasing there is no surprise (with increasing the file size).
Write to file and close time are almost constant. I thought to open file in setup() and initialize timer. Write to file in loop() and after some time interval to close the file, after that open new file and so on. But as can be seen the print operation dataFile.println(dataString); takes approximately 2 ms that is 500 Hz, and it is too slow.
I'm trying to find way to save data fast.
I'm new to arduino and don't know all capabilities and I ask for help.
I have some thoughts how to be:

  1. Sounds crazy to connect to PC with 30 m usb cable. There is potential risk of weak signal.
  2. May be use wifi or bluetooth to transfer data? I did not do it and afraid there is some limitation as with SD card
  3. Connect the arduino to another and save Serial. Is it legal?))
  4. Find way to efficient save file on SD card. May be save in buffer binary data and to save binary once a some period.

Any suggestions are appreciated.

UPD

  1. For 500 Hz acceleremeter I use Arduino Uno
    For 5 ะบะz I use Arduino Due
  2. I want to store approx 30 min of reading. I can save data in different files let's say with 2-10 sec of recording
    I checked that if I write with 5 kHz 1 s of recording will weight 110 kB if it writes floats

You don't say:

  1. What processor you are using, or
  2. How much data you actually want to record

An ESP operates faster and has more memory than a plain-vanilla Arduino.

If you have enough of it, you might save to an array in on-board memory, then transfer at leisure.

With an Arduino, and possibly with an ESPxx, opening before and closing after a print to file is not only, clearly!, "time-expensive", it is also bad practice and will probably lead to grief later if not sooner. Open the file in setup and leave it alone. I don't know if that will save your bacon, but it will go some way towards it.

You might also check the data storage method. I don't think floats are very efficient.

Also, making each write subject to a test can't possibly be a good idea. Once you have confirmed the SD is properly in its slot and working, in Setup, you should be reasonably confident that it will still be there for the loop.

As your units are arbitrary you can easily use integer values by changing the scale factor and doing the math yourself rather than using the mapf(

There are USB cables that have a optical fiber inside or with twisted pair ethernet-alike signals. If both ends of the USB cable are devices with their own power supply, then a USB cable of 30 meters is possible. USB 2.0 is good enough for a Arduino board. I have a 10 meters USB cable with optical fiber and never had a problem with a Arduino board at the end, not even when it is powered only via that USB cable. My computer sees the USB cable as a extra USB hub.

1 Like

There are several ways to save data from a sensor attached to an Arduino. If you're connected to a personal computer, you can simply send the data from the Arduino to the personal computer serially, and save it to a file . If you've got an SD card attached to the microcontroller, you can save the data to the card.

Never heard of those before, interesting! But do the cables cost more than the Arduino they connect?

I got mine for free, it was a gift :smiley:
They are sometimes called "optic" usb cables. I see there are cables up to 50 meters for USB 3 already. The 5V and GND wires are still normal copper wires and have a voltage drop of course.

Thaks for replying, I added UPD

Thanks for sharing info. It's great idea, never have heard about such cables before

Never heard of that before...
16 feet for $147
(?)
https://www.ebay.com.au/itm/275194522284?hash=item4012df2aac:g:3u4AAOSwI7ViIlqp

Look at the teensy4.1 for the logging speed you need.

See: Need for speed ยท Issue #372 ยท greiman/SdFat ยท GitHub (the need for speed)

1 Like

Hi community.
I have an accelerometer adxl377 and want to record acceleration with it. I connect adx377 to arduino due with 3.3v, ground, a0, a1 and a2 inputs.

Here is my sketch:

int scale = 200; 

void setup()
{
	// Initialize serial communication at 115200 baud
	Serial.begin(115200);
 
  analogReadResolution(12);
}

// Read, scale, and print accelerometer data
void loop()
{
	// Get raw accelerometer data for each axis
	int rawX = analogRead(A0);
	int rawY = analogRead(A1);
	int rawZ = analogRead(A2);

	float scaledX, scaledY, scaledZ; // Scaled values for each axis

     scaledX = mapf(rawX, 0, 4098, -scale, scale); 
     scaledY = mapf(rawY, 0, 4098, -scale, scale);
     scaledZ = mapf(rawZ, 0, 4098, -scale, scale);


	// Print out raw X,Y,Z accelerometer readings
	Serial.print("X: ");
	Serial.println(rawX);
	Serial.print("Y: ");
	Serial.println(rawY);
	Serial.print("Z: ");
	Serial.println(rawZ);
	Serial.println();

	// Print out scaled X,Y,Z accelerometer readings
	Serial.print("X: ");
	Serial.print(scaledX);
	Serial.println(" g");
	Serial.print("Y: ");
	Serial.print(scaledY);
	Serial.println(" g");
	Serial.print("Z: ");
	Serial.print(scaledZ);
	Serial.println(" g");
	Serial.println();

	delay(2); // Minimum delay of 2 milliseconds between sensor reads (500 Hz)
}

// Same functionality as Arduino's standard map function, except using floats
float mapf(float x, float in_min, float in_max, float out_min, float out_max)
{
	return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}

I put the accelerometer on the table, so I expect to have x~0 y~0 and z~1 g.
But there is the output:
The output is:

X: 2047
Y: 2032
Z: 2003

X: -0.20 g
Y: -1.66 g
Z: -4.49 g

X: 2046
Y: 2043
Z: 2047

X: -0.29 g
Y: -0.59 g
Z: -0.20 g

X: 2000
Y: 2039
Z: 2049

X: -3.90 g
Y: -0.59 g
Z: -0.29 g

X: 2052
Y: 1978
Z: 2050

X: 0.29 g
Y: -6.93 g
Z: 0.10 g

X: 2039
Y: 2043
Z: 2059

X: -0.98 g
Y: -0.59 g
Z: 0.98 g

X: 2048
Y: 2037
Z: 1973

X: -0.10 g
Y: -1.17 g
Z: -7.42 g

X: 2012
Y: 2085
Z: 1988

X: -3.61 g
Y: 3.51 g
Z: -5.95 g

So there are some random peaks on all analog inputs x, y and z.
There is strange because when I work with arduino uno with the same accelerometer there are no peaks. There is some bias of output (here is my question what to do with this https://forum.arduino.cc/t/recording-acceleration-with-adxl377-and-arduino-uno/1012697) but I can deal with it.

I tried to change analog resolution to 10 bit and different scales like this:

 analogReadResolution(10);

scaledX = mapf(rawX, 0, 1023, -scale, scale);
scaledY = mapf(rawY, 0, 1023, -scale, scale);
scaledZ = mapf(rawZ, 0, 1023, -scale, scale);

But peaks of g do not disappear.

Tried to change delay to 200: delay(200);

I can not understand why this is happen.

P.S. I do not touch the accelerometer it lies on the table with no vibration or impacts.

That +/- 200 g accelerometer is for high impact measurements, and 1 g is around the noise level. You might be able to measure 1 g if you carefully calibrate the sensor, and remove the offsets along each axis.

See Calibration and Programming | Adafruit Analog Accelerometer Breakouts | Adafruit Learning System

@kracozebr,

Your other topic on the same subject deleted.

Please do not duplicate your questions as doing so wastes the time and effort of the volunteers trying to help you as they are then answering the same thing in different places.

Repeated duplicate posting could result in a temporary or permanent ban from the forum.

Could you take a few moments to Learn How To Use The Forum

It will help you get the best out of the forum in the future.

Thank you.

Now I found your other topic on pretty much the same subject, so I have merged them.

One more duplicate topic gets you a ban from posting for a few days.

Thank you.

Hi @PerryBebbington The topics are different, when I post several questions in one topic the moderators make comments to me that it is necessary to separate questions

If it's all about the same project it's best to keep everything in one topic because the answers to one question provide the context for the next.

If you add your question on the end of someone else's question that's hijacking their topic. If your question really is the same then you don't need to add it as it's already been asked and answered. If it's not the same then ask in your topic.

If I'm still not clear please ask me to clarify.

If you are not happy with my decision please flag this reply to the moderators and ask for one of the other mods for their opinion, I will respect what they say.

Actually I found the solution for task of logging analog inputs with sdfat library ( There is the link to example of logging sketch github: SdFat/examples/AvrAdcLogger at master ยท greiman/SdFat ยท GitHub ) and sdfat adafruit fork- https://github.com/adafruit/SdFat/tree/master/examples/AnalogBinLogger. But I have an issue none of my 4 sd cards can not be used. I get error: "check sd format".
I opened an issue on github but the developer do not know why there is the issue - SD card is not initialized ยท Issue #391 ยท greiman/SdFat ยท GitHub
Has anybody been working with these libraries? May be one knows how to correct format the sd card to work properly?

Finally I with the developer find solution of issue with no reading sd card. One interested should follow the link to the issue: SD card is not initialized ยท Issue #391 ยท greiman/SdFat ยท GitHub

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.