Does the next line of code not get executed until the library finishes writing to the SD card? Or does the "file closing" take place in some interrupt routine?
The reason I ask is that the SD spec says to allow for up to 250ms (that's huge!) for finishing writing a block of data. If it does actually take that max allowed time then there's going to be a lot of missing data from buffer overflow. Since 250ms of 9600 baud data is about 240 characters and the serial buffer is only 64.
It depends on your next line of code. If your next line of code is "dataFile.open();" the answer is Yes it will wait. As for the wait time, that depends on the SD card. A write latency time of 250 ms is the maximum time I have ever recorded for an old 2MB SDSC card. A typical latency time for a 16-32GB SDHC card is 18-36 ms. Another factor affecting write latency is the number of write/erase cycles to a block or cell. After a few hundred write/erase cycles or 10-20 format cycles latency times increase to the 30-40 ms range and at that point I retire the SD card from high speed datalogging.
Okay, so if it waits when the next line of code is "dataFile.open()" then the file closing must be taking place in an interrupt behind the scenes?
I'm still having a hard time with the 250ms timeout. Sure, cards do write faster than that. But it seems like we're all taking advantage of that and not programming for the worst case, but that would require a huge buffer. I guess your method of retiring slow cards is a good approach.
I've never seen any data on it, but in theory there may be a big difference in write speed if the SD card is fully erased before each logging session. The SDFat library has an example called SDFormatter which erases the entire card before doing a quick format. I don't know if the recommended formatter from the SD Association also does that - its option is to do an "Overwrite" format, which may not be the same thing at all. I believe, but don't know for sure, that cameras which offer a "low level" format option are erasing the card so as to improve the burst speed.
The benefit of pre-erasure is that the card's controller will not have to erase anything, or move anything, prior to writing to the flash memory. It could be that process that is responsible for the occasional long write times.
The other thing that takes up time is updating the FAT every time you need to write to a new cluster. If the FAT update requires erasure and wear leveling moving of data, that could be expensive time-wise. To get around that, you could create a file in advance with a filelength of, say, 2GB, and manually assign clusters to it in the FAT assuming the file will be written in consecutive sectors. Then the actual logging would need to be done with low-level commands to the card, such as a multiple sector write, without bothering with any of the FAT overhead. Then at the end, you might want to update the filelength to reflect what you actually wrote to the file. I'm not aware of any library that works this way. You are basically just writing data "raw" to consecutive sectors of the card without any consideration of the file system.
I ran across some other write latency issues in the forum and I took my 400 Hz datalogger and ran it for 6 hours last night generating a 114 MB file. The write latency time at the beginning of the file was 45ms, 6 hours later the write latency time was 472ms. The longest I have ever recorded. Normally, these dataloggers operate for a few minutes during an amateur rocket flight. New SD cards have write latency times in the 18-20ms range. I know that formatting and write/erase cycles affect latency times, but a 472ms latency time from continues operation is not acceptable. I need to understand this change in write latency times if I'm to use these dataloggers in other applications over extended time periods.
Well that's a horrendous delay. But I still wonder if the non-erased state of the card is responsible. Are you able to repeat the test with an erased card? You could low-level format the card in a DSLR camera, or install SdFat.h, and format the card with the SdFormatter example.
I don't believe a card gets slower with usage. I think it's just that a brand new card is always fully erased. If an old card is fully erased, I think it should perform like a new card.
I'm not talking about regular formatting, or overwriting the card with all ones or all zeros. I mean literally erasing the flash memory.
Ran another 6 hour data collection with the same SD card formatted with the SD Association formatter using the overwrite format. The file generated was 162 MB. The write latency was 45ms at the start of the file increasing to 178ms at the end. Significant improvement, but only marginally acceptable. Does look like a SD card overhead issue within FAT.
16 GB card. Ran another 7 hour test with the previous file renamed so I had two files on the card. Very similar results starting at 45ms and ending at 202ms after 7 hours. I don't have a circuit analyzer or even a good oscilloscope, but the LED on pin13 showing the SCK and another LED programmed to show the data reading loop, indicate that my program is looping through the sensor read and SD write at the programmed rate. The SD card is not accepting the data and the SCK stops cycling and stays HIGH. Appears we have a busy signal being read either in the SPI or SD library.
Getting back to MK1888's original question. The next line of code is executed after the dataFile.close(). If the SD card is busy at the next dataFile.open() action it will not be accepted and the program falls through to the next action.
I'm still attempting to get my oscilloscope running stable, after 26 years sitting in the closet, to see details of the SPI lines.
I also found that at slower SPI clock speeds, the recorded write latency is reduced. I don't understand that effect. Can anyone explane?
Spacejohn, would you be willing to post the code you're using for these tests?
Also, I don't understand why the delays always start small and get bigger. I assume even a 200MB file is nothing for a 16GB card. So what is happening? Is the card getting hot, and slowing down?
We are not looking at the correct end of the problem. My oscilloscope shows no change in the time period for dataFile.close() to run. It did show a slow drift for SD.open("Data.txt", FILE_WRITE) to run each time it was called. The time drift is not SD card write latency, but an SD function opening latency.
I've installed the SDFat.h library, and have been playing with the examples. The first one was SdFormatter. Looking at the code, it clearly is erasing the entire card using the SD card's flash erase command. It also reads in a register flag from the card which indicates the "erased state" of the memory - whether erased bytes are zero or FF. This varies by vendor. On my Transcend card that reports back FF. Then when I examine the card using the hex editor on my computer, it is indeed filled with FFs.
Then I formatted the same card using the formatter from the SD Association, selecting the "overwrite" format option. While the SDFat formatter erased the entire card in a few seconds, the Association formatter was very slow, and after 10 minutes I gave up at 50%. Then looking at it with the hex editor, I found the first half of the card was all zeroes, and the last half was still the erased FF state. So this confirms my suspicion the Association format is literally overwriting the entire card with zeros, which is why it was so slow. More important, it is leaving every byte in the written state, with no remaining erased sectors. So in effect, after formatting, the card is completely full, which is the opposite of what you want.
Then I did a "low-level" format of the same card in my Canon DSLR. That took a few seconds, and returned the entire card to all FFs. The camera needs the card to be erased in order to maximize the frame rate in burst mode.
I think the Association utility is really terrible, and I don't think anyone should ever use it except maybe for a Quick format. It doesn't take much to install SDFat, and if necessary dedicate a Nano and SD module just for formatting cards. The sticky post at the top here needs to be revised. It should not be telling people to use the Association formatting utility.
Here's the log of the SDFat SdFormatter operation:
Type any character to start
This program can erase and/or format SD/SDHC cards.
Erase uses the card's fast flash erase command.
Flash erase sets all data to 0X00 for most cards
and 0XFF for a few vendor's cards.
Cards larger than 2 GB will be formatted FAT32 and
smaller cards will be formatted FAT16.
Warning, all data on the card will be erased.
Enter 'Y' to continue: Y
Options are:
E - erase the card and skip formatting.
F - erase and then format the card. (recommended)
Q - quick format the card without erase.
Enter option: F
Card Size: 8166 MB, (MB = 1,000,000 bytes)
Erasing
................................
.............................
All data set to 0xff
Erase done
Formatting
Blocks/Cluster: 64
FAT32
................
Format done
Setting a number of write blocks to be pre-erased (ACMD23) will make a following Multiple Block Write operation faster compared to the same operation without preceding ACMD23.
Yes I think the SdFat logging example uses ACMD23 to pre-erase the blocks. But the formatter example uses CMD38 to erase. I don't know if there's any difference in the result.
I thought my PIC micro, showing 80ms to write data/directory/FATs, was high. I had worried about losing incoming data from the GPS. The math said it was a given. And, in fact, I confirmed it by comparing the file on the SD card to the actual GPS output.
I wondered...How can the Arduino work so well? I'm running the PIC at the same speed! So I hooked up the oscilloscope to the Arduino and got a whopping 120ms. Turns out, of course that too is losing data.
I made some changes to my PIC code and got down to 65ms for writing a data block, updating the directory block, and updating the two FAT blocks (FAT1 and FAT2). Things I changed: Bumped up to 32MHz via on-board PLL (not sure you Arduino folks can do that). I used indirect addressing for copying data from the microcontroller buffer to the SD card. And I upped the serial buffer to 80 chars...The Arduino serial buffer is only 64 chars.
My next step may be to write in all in assembly. I really want to get the time down so I can add an OLED display.
MK1888, I would urge you to look at the "AnalogBinLogger" example in the SDfat library. He creates the file in advance, and erases it with ACMD23, and from that point he just does a multi-block write to consecutive blocks without doing any file system updates until logging is terminated. So while logging is going on, nothing is happening with the directory entry or with FAT1 or FAT2 to slow down the process.
He also uses two 512-byte buffers in the Uno/Nano. One is SDFat's own buffer, and the other is created for this process. He uses a timer interrupt to collect the samples at a fixed interval and save them to the current active buffer. Then when that buffer is filled, he switches to the other buffer.
Then back in the main loop, he notices when a buffer has been filled, and then writes that to the card as part of the stream of multi-block writes. So that means that no data will be lost so long as the write doesn't take longer than 512 interrupts.
I ran this example with the default 5000 samples per second, and it worked flawlessly. The biggest issue is likely to be having enough ram. The two buffers take up 1024 bytes, which is half the ram on a 328P. So the remaining 1024 bytes is all you would have for the heap, the stack, and whatever else is needed. It would just depend on how complex your code becomes.
Here's the log of my 20-second sample run:
FreeStack: 1135
type:
c - convert file to csv
d - dump data to Serial
e - overrun error details
r - record ADC data
[r]
Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
[x]
Truncating file
File renamed: analog00.bin
Max block write usec: 1004
Record time sec: 20.130
Sample count: 100642
Samples/sec: 4999.60
Overruns: 0
Done
type:
c - convert file to csv
d - dump data to Serial
e - overrun error details
r - record ADC data
I did try that route, and it brought my time down considerably. I can always fall back to that. Getting rid of directory and FAT updates I can get my "write" time down to under 20ms for single block writes. That certainly gives me plenty of time left to do OLED updates. And there are ways to bring overhead down quite a bit, if you take a look at Petit FAT by Elm Chan. I'm trying to see how far I can push it and still allow other files to exist on the card. I might use this for bike rides and hikes so I'm not fond of having to copy a file off the card all the time, or overwriting or appending. It's interesting to figure out how to schedule a bunch of things to work together.