I’ve been creating a data logger that I’m going to use when creating new projects and/or add them to existing projects to understand why long term failures happen.
To test it, the ESP32 on the left sends a data string every 2 seconds, using the TX & Rx pins, to the ESP32-C3 on the right and the ESP32-C3 writes that data to the micro SD card in the middle. It works perfectly but I need to know what is the correct way to log data onto the SD?
Each line of data is received and it opens the log file writes the string and then closes the file.
I pulled the card during the write cycle and the log file became corrupt and my Mac would not open it. So, if I open the file and continually append it and I lose power the file will become corrupt. If I continually open and close it, I might miss serial data. Also, when the file becomes large, it will take longer to open and close it causing data loss.
Do I wait for set intervals? Like open the file and continually append it for a number of seconds and then close it and then reopen it again?
Do I continually add to the string and then when it gets to a set size open the file and write to it then close it?
Do I create different log files on a daily basis so if one become corrupt the rest will be OK? That said, if the issue I’m trying to log causes a crash then the file would become corrupt anyway because a file closure didn’t take place.
I have no idea what the correct process is. when logging data to an SD. Any help would be appreciated.
Power fails? Maybe you need a battery for backup. Also, if you do this, the device will also need a charger and a voltage sensor to avoid logging data when there is no power and battery is about to die.
Another aproach could be logging the data into the ram of the chip, when its about to be filled start saving data to the SD (open file, save data, close file). If the file is large anought (24hs data or something like that) create a new file. I assume your program checks the time via wifi, this could be done easily checking the current date.
I think the safest way to log data to an SD card was shown in an SdFat.h library example called LowLatencyLogger. But that example has disappeard from the latest version of the library. I don't know if that was intentional. Perhaps @fat16lib can tell us.
Anyway, that example completely erases the card, then creates a number of files of 128MB each (as I remember), essentially filling up the card with those files. It doesn't write all that data to the card, but just manipulates the root directory and FAT tables to create the files. The actual contents of the files is whatever state erased sectors read as - either zeros or FFs. The files are created such that their data sectors are sequential and contiguous, so you don't need to access the FAT to find the next sector or cluster.
Then the logging process is to go to the first sector of a file's data and begin writing directly to the card's sectors, and continue doing so until the file is filled. Then you go to the beginning of the next file.
If something crashes, nothing is lost, and no files are corrupted. That's because the files were pre-defined, and are never opened or closed after first being created. Of course you are always at risk on up to 512 bytes of data because you can only write to the card's flash in 512-byte bundles. Actually, you can send data to the card one byte at a time, but the card's controller will save up what you send until it has 512 bytes.
I don't know of a faster or more reliable way to log data to an SD card. You do have to do low-level commands to the card, which the FAT file system normally lets you avoid, but the SdFat library lets you do that.
Arguments about this have been going on for ever, will continue for ever, and using an ESP32 probably just adds to them. But, no matter what, starting a new file at midnight and using date as filename, must always be a good idea.
using microcontrollers to acquire data for transmission to remote servers, e.g. over Ethernet, WiFi, BLE, LoRa, etc
generally would use Interrupt service routines to acquire binary data from sensors storing data into double buffers - when a buffer is full functions in the main code would transmit it to the remote server
clearly one has to check for overruns on the ISR data acquisition and on the double buffering
currently looking at logging Canbus data to SD card
Hi Horace, funny you should mention CANBUS. I have just bought a new car and was looking at the CANBUS modules myself on AliExpress . Being a software developer, I’m going to create an iOS app that automatically connects to the ESP32 and drag the data off either live logging or historic downloading after the wife has used it. I'll probably add a GPS module to it also.
I’ve just added another post, on a tangent to this post, regarding on how projects can expand and transform. I got the data logging working but mutated it into a simple Serial Monitor & Data Logger. Because I won’t or don’t expect streams of data to log, I decided to keep the ‘open, write and close’ method per line of data to log. Some won’t agree with this method but it seems to work quite well so far.
. Being a software developer, I’m going to create an iOS app that automatically connects to the ESP32 and drag the data off either live logging or historic downloading after the wife has used it. I'll probably add a GPS module to it also.
Could you take a look at this and see if you could resolve it for me please?