Best way to optimize Arduino sensor data logging without losing readings?

I'm working on an Arduino project that reads data from multiple sensors and stores the values on an SD card for later analysis. As the number of sensors increases, I've noticed occasional missed readings and delays in logging. I'm currently using standard SD card libraries and reading sensors sequentially.

What are the best practices for improving data logging performance on Arduino? Would using interrupts, buffering data in RAM, or switching to a different storage method make a noticeable difference? I'd appreciate any recommendations from members who have built reliable long-term monitoring or data acquisition projects.

I moved your topic to an appropriate forum category @techy247.

In the future, when creating a topic please take some time to pick the forum category that best suits the subject of your topic. There is an "About the _____ category" topic at the top of each category that explains its purpose.

This is an important part of responsible forum usage, as explained in the "How to get the best out of this forum" guide. The guide contains a lot of other useful information. Please read it.

Thanks in advance for your cooperation.

1 Like

Without more information about the project (for example which sensors we're talking about, number of sensors, frequency and duration of sensor reads) and the code (are the sensors read sequentially or at different times? And how often do "occasional" missed readings occur?), it's difficult to tell you how to improve logging to avoid overwriting the sensors.
If sensor readings are timed, you might plan to keep the data in RAM until you're sure you don't need another reading.
I'm afraid I can't say more without more information.

Given the lack of any detailed information on your hardware or code, I'm going to guess it's poor code.

Does your code open, append data and close the SD file for each sensor reading, for example?

If so, as the length of the file grows, the time taken, to open the file and read through it to find the end before writing new data, will take longer and longer.

I'd also check whether the SD card writes are blocking the main loop for too long. Buffering a few readings in RAM before writing them to the card has worked well for some of my projects and helped keep the sensor timing more consistent.

Thanks! To answer your questions:

  • Sensors: [e.g. "3x DHT22 + 1 pressure sensor"]
  • Reading frequency: [e.g. "every 500ms"]
  • Reads happen sequentially, then written to SD right after each one
  • Missed readings occur roughly [e.g. "every 30-60 seconds"], no fixed pattern

Would buffering readings in RAM and writing in batches help here? Happy to share code if useful.

Thanks, that's helpful! I'll check how long the SD writes are actually blocking the loop and try buffering a few readings in RAM before writing, like you suggested. Will report back once I've tested it.

But when is the file opened and closed? For each individual sensor? Or once for each 500ms? If the former, I would certainly recommend trying the latter.

Which one?

Takes between 5-6 milliseconds to read per sensor.

Can be read once per two seconds, so your reading frequency of 500 ms might be a recipe for problems.

Note the reading of the DHT22 sensor (protocol) is timing sensitive so you need to disable interrupts before reading them to prevent low level problems.

If you need persistent buffering, you might consider using FRAM instead of RAM.

I would use a software FIFO queue (implemented as a static circular buffer).

Sensor readings populate the queue asynchronously, via interrupts or dedicated tasks (if possible) preventing lost data during SD card write latencies.

Writing CSV records to the SD card then pops elements from the queue, ideally in buffered blocks to minimize wear and power consumption.

If you don't open and close the file after every update, the SD card library will buffer a 'page' of data in RAM automatically for you, I think, before writing it to the SD card. Suspect a page is 256 bytes but not sure. The danger with this approach, as with all approaches using buffering, is that a power failure or other failure will result in some lost data.

We did ask! :wink:

512 bytes, Same as Microsoft MSDOS floppy disk.

Reading the DHT22 every 500 ms is outside the sensor specification.

The manufacturer’s AM2302/DHT22 datasheet specifies:

“Collecting period: 2 seconds”
“Collecting period should be > 2 seconds.”

DHT22.pdf

Therefore, each DHT22 should be read no faster than about once every 2 seconds (0.5 Hz). Reading it four times faster can cause stale values or communication errors.

See @robtillaart's post #9.

Edit:
What exactly needs to be measured, and why does it need to be sampled so frequently?

I suppose if each of the 4 sensors is read in turn, with 500ms interval, then each individual DHT22 would only be read once every 2 seconds. Maybe that's what @techy247 meant to say.

No code or wireing diagram. We can't compile word salad.

Okay, so do you mean you read ALL 4 sensors every 500 ms or one at a time? (you'd avoid answer this question if you published your code...)

We're talking about temperature, humidity, and pressure: why on earth do you think these values ​​can change so quickly? What are you monitoring exactly, a Mars rover?

I don't think you need that frequency; a 5-second sampling rate is usually sufficient.

Additionally, if you save the entire data batch to the SD card at the end of the polling (I hope you don't open, write, and close the file for each sensor/file...), you're pretty sure you won't encounter the problem you're complaining about.

However, more information about the project scope together with a diagram and the source code would allow us to better assist you...

Don't sample your data faster than you can write it to the SD card.

Using a RAM buffer is usually a good first step, since SD card writes can introduce unpredictable delays. Collect sensor readings in memory and write them to the SD card in batches, while using interrupts or timers for time-critical sampling. Also consider pre-allocating files and avoiding frequent flush()/open-close operations to reduce write latency.

None of that matters if you are sampling data faster than you can write it.

Other points for reliable sensor logging

  1. add a sequence number per log entry to detect missing samples
    this might be a timestamp
  2. add a CRC-16 per log entry to detect failing bits - or alternatively add a CRC-16 per SD block write to detect failing bits.
  3. use compression, see below
  4. do not put all data in one file on the SD card, use e.g. one file per hour and a folder per day. If the card gets corrupted, the chance that you can save partial data increases. Or a new file every M records.

Compression

  • use compression, e.g. run-length compression of identical measurements.
  • write raw sensor bytes, DHT22 reduces 50% (pressure sensor unknown).
  • use delta coding, as values do not change that much (assumption) you might only need only to store the deltas. With one byte you can encode +-12.7 degrees with respect to previous sample. (75% reduction).

Yes, compression takes time, more code which can go wrong, needs decompression however in general it enhances the throughput and could roughly double the storage capacity.