I'm working on a project where I need so save data from multiple sensors on a SD card. I don't want to save the data of all the sensors with the same frequency, and the final goal is to plot their curves. I will have up to 32 times the same sensors gathering 2 different (Pmax(t) and I(V) curves) data each and 2 other sensors (temperature(t) and Lux(t)).
The device will be used for ~500 hours measurement campaign
My problem is that I don't want to have many different files and I read opening and closing too quickly and too many times could lead to issues.
So my question if how can i write on file (up to 2) efficiently (according to SD specifications) so i can more or less easily plot that curves.
What I was thinking of doing is like all the data has 2 variables, I write everything on 3 columns the first as an identifier allowing then to filter and sort, the 2nd column DataX and the 3rd column dataY
Have you computed the total number of bytes necessary to store for each of your scenarios?
How often are you sampling the sensors? Remember there is a lock-out time for your program when ever you fill a 512 byte block an the data if physically written to the SD card. No sampling can take place during that time period.
You need to do a "flush" occasionally to ensure the SD card directory is updated with the current file length.
And you need a way to tell your program to close the current file when you are ready to shut the project down.
Unless you have created CSV files in the past, remember that ALL values are ASCII characters, not Arduino integers, etc. So, you need to decide if you will used fixed length fields in the CSV record, padded on the left with zeros. Or will have variable length fields with no padding of zeros.
You can have a file open on SD Card, you can write your values into a file.
As "Paul_KDB7HB" has mentioned:
as long as the file is open for write in your MCU - you cannot take SD Card out!
you have to close the file (and flush all pending writes, let the FileSystem close properly) before you can take the SD Card.
What you write into a file - is up to you: text, a CSV file format, hex, binary, ... all up to you.
Just to remember:
as long as SD card file is open and "in use" - just close all before you remove SD card
even when closing a file properly - I would make sure that also SD card as device is "unmounted" (it can still happen that file is closed but the SD card buffers are not flushed, not written)
You can try: potentially you get a corrupted file system and all data on SD card is lost (all files are corrupted). An external storage device needs careful handling "how to release the device" and how to make sure that all data, also cached data, like directory entries, are flushed/written.
Maybe you know how to unplug a USB stick on a Windows OS system. Here, it is the same issue: the the MCU, FW - you want to remove the SD card.
Performance wise:
writing files to SD card works fine, but if you try to write faster as the SD Card Interface can transfer to external SD Card - and also think about the overhead due to the FileSystem - you lose data (or you are unable to write).
Consider an SD Card as a very slow device to record your data.
And do not forget to close the written file, to "release" the SD card (before you take card away).
Im working on a prototype with one sensor, with one sample of one point every minutes for 500 hours and 19 samples, of 120 points, spread on the 500 hours, probably float. What i intended to do is flushing every hours and after each of the 19 sample and close after the 500 hours.
Can you explain me how work the "512 byte block" (cached data ?)
And i dont understand what you mean
Unless you have created CSV files in the past, remember that ALL values are ASCII characters, not Arduino integers, etc. So, you need to decide if you will used fixed length fields in the CSV record, padded on the left with zeros. Or will have variable length fields with no padding of zeros.
512 bytes is the sector size: this is the block size as any data is written. Even you write just few bytes - on SD Card it has to write always 512 bytes as one single block (one sector).
So, in order to avoid to write for every single byte updated again and again almost the same, instead always a completed 512 byte block - the FileSystem has a cache: it will update when the 512 byte is full (or even larger block sizes possible).
It means also: if you write bytes but a block is not yet completed - it sits in cache memory, not written yet. Therefore flash and close the file properly.
What you write is up to you. The comment was just: a CSV file is a text file: you have to write text, not binary values. So, your internal values are converted into text before writing into a CSV file (opened as text file).
So the SD automatically « flush » as soon as it cache is full (wich mean 512 bytes written).
And exept taking more time, what cause flushing every time i write something (on the case every data is important and i have enough time)
You do not know when a sector is flushed - it is up to the FileSystem.
I can be that FileSystem decides to flash after time elapsed when sector was not written anymore.
Or: if a new sector is needed - maybe flash the old one first.
But FileSystem can also assume you want to go back write again into the current sector.
Do not take anything as granted if and when flashed: just make sure to close your file when done (before you remove SD Card). You can force a flash via function call, even FileSystem might flash after a while automatically.
The problem with "flush" and "close properly" is not just related to writing sectors!
Every file has a directory entry, somewhere else on SD Card. If you write to file, it is open, the size increases. So, this file size must be part of the directory entry. This directory entry, updated all the time, sits also in a cache.
Even the sector was written - the directory entry was not written! As long as the file is open, the directory entry sits in cache memory. This will not be written until you close the file.
The FileSystem does not know if you will keep writing and increasing size, so that an update on directory entry is still needed. Why written "too often" (and wasting time, reaching the max. write cycles possible by SD Card etc.).
Don't care, just: close the file, unmount the SD Card before removing it.
If you want to be on the safe size: call flush function when something has been done.
But it might guarantee that also the related directory entry was written - but who knows for sure?