Optimised setup for saving to SD, using large buffer, low current, low risk of corruption etc

Other Hardware Storage
1

A bit difficult to write the description there.

I want to log at extremely low battery power about 250 bytes per second but using an nRF52840 which has 256k ram, so I am assuming I should have at least 128k available to me for a buffer. Meaning I can save my 250 bytes per second data and write a 128kb chunk of data every 800 seconds which I assume is more efficient battery wise.

Powered by a coin cell so as low power consumption as possible is essential, what I am wondering is whats the best way to deal with the file system? I would ideally like to be as 'data safe' as possible so that if I do unmount the file (like via a power cut even) I can recover as much data as possible.

I am thinking probably one single file is fine, there seems to be little advantage to splitting it up into one file per write, but is this right?

And can I just SD.open the file, and write to it, then sleep for a while (which also means sleeping the SD card), and then wake up 800 seconds later and wake the card up and write to it again? And if I can do this is it actually the best way to do it?

OTOH I could close() the file each time I go to sleep and sleep the SD (to be clear I'll be sleeping the nRF52 at 250Hz I guess between ADC conversions, but sleeping the SD for 800 seconds at a time while I refill the 128k buffer with ADC data).

I'm familiar with wear levelling, the large block erase that SDs perform, etc, so it seems to make sense to try to write some multiple of the erase block size each time so that it does not keep copying a block to each time I add to that block. So hopefully I can find an SD with a erase block 128k or less. Whether this is something i can detect programmatically IDK.

I'm hoping @fat16lib might comment as I think he wrote the library I am going to use and some of these questions I think concern the inner workings of that library.

But also if anyone knows what consumes the least power (like, does it take more power to keep opening and closing the file each time, or does it make no difference) I would really like to ask that too.

If the card is removed during the sleep period between writes while the file is still open does this mean the data is not recoverable or does it not make much difference? I am prepared to read the SD with a custom reader, even through the same nRF52840 device if necessary and send the data to a PC via cable or bluetooth. Its more a question of how the handling of files works and if they will be very broken if they are not closed etc.

I've looked into using a serial data chip BTW but I need about a GB or two of storage and it does not look that cost effective.

Tia, Pete

2

In previous versions of SdFat.h, Bill (@fat16lib) had an example called LowLatencyLogger. It created multiple files on the SD card ahead of time, containing contiguous sectors, with all those sectors being erased. Then during logging, you just write directly to the sectors, completely ignoring the file system. You don't ever lose anything that has been written to the card because the file has already been created at maximum size, so you are effectively writing data into sectors the file already contains.

But apparently this method no longer works with newer versions of the library. I don't know why. But the IDE's Library Manager will let you install earlier versions. I know the example is included in v1.1.2, but I don't know the latest version which still contains it.

3

Have you taken a look at FRAM, you can update it byte by byte and it does not draw much current. FRAM was designed for ultra low current projects.

4

@gilshultz I had never heard of FRAM and it looks really interesting but for this project probably too small, the largest I can see is 16Mbit but really I need 1 to 2 GB. Still I think I'll get a couple of these in my next order and have a play with them, they seem to have some interesting advantages thanks.

5

An SD card can draw over 200 mA during a write operation. Coin cells aren't designed to deliver peak currents nearly that high, and if the idea works at all, battery life will be severely impacted.

sleep the SD

They do that automatically, when not actively reading or writing.

6

@ShermanP thanks, very interesting. I do wonder why it no longer works, maybe if Bill comes by here he can say. Sounds like a particularly useful function for this community since a lot of what is done here involves batteries, logging etc.

Anyway it made me think, in fact I don't need a file system anyway. I am happy to plug the card into a linux system and dd the data off, for instance, or even use the nRF52 to read the card and send it over USB. I've done a bit of coding to make a linux file system module that read and wrote sectors and it was not difficult, each sector has a number and is fixed size and thats all there is to it, you just read and write them based on number etc.

If directly writing sectors and ignoring the FS is possible why not do that in my case. I will take a look at this older project as you say and see how its done, and maybe be able to copy the relevant code directly even. I guess if its 'not possible' now in the latest versions there must be a reason, something at a lower level has changed maybe and that prevents it, so it might not be as easy as all that but its worth looking.

7

FRAM chips come in various sizes, typically ranging from 4 kilobits (kb) to 1 megabit (Mb), with common densities including 4, 16, 64, 128, 256, 512, and 1024 kb. Larger sizes, such as 8 megabits, are also available, but the specific size can depend on the manufacturer and the application.

The ones I use are 32K x 8 on a 8 pin SOIC. They are on I2C but SPI is also available. They work at memory speed and are byte addressable. There is no delay when writing or reading so you could use them for the buffer and when written it is in non volatile memory. You can transfer the data whenever you want, power fail will not be a problem. I got a bunch from my favorite china supplier, very inexpensive. I typically replace the EEPROM on the RTC module with them. No same I2C address.

8

@anon94801460 yea I did wonder about the peak currents taken by the SD. I wonder though if its really brief (a few ms) maybe a decent capacitor could carry that peak. Obviously fine tuning the power supply is a project in itself but my initial instinct would be to use a decent step down regulator of the right kind of rating (IE very low power) and use 2 cells for 6v, capacitor at 6v with a drop out of 1v and a system voltage of 3.3v gives 1.7v headroom for the capacitor to fall during the SD on period. As an example in that system, 1000uF would supply 200mA for about 10ms. Very useful that the cards automatically sleep - no need for any commands at all? Just automatically wake up when you write?

9

Yea as you say since they can be written so many times, perfect as a larger buffer and means I would not be risking the data in volatile ram. Byte writes are hugely useful too. I will definitely get a few of these interesting sounding chips.

10

Let us know how it works out. It will probably be more cost effective in the end as well.

11

Yes, but I still need the SD unfortunately as I need to save a GB or two. But the buffer aspect is really interesting.

12

I doubt very, very much that you can make a reliable SD storage system with a coin cell. Any voltage sag will lead to fatal errors, and battery life will be extremely short.

In any case, the Arduino SD library is very error prone even in the best of circumstances, orders of magnitude less reliable that other implementations, such as the RPi.

13

@anon94801460 They take that much current do they? I was prepared for some high power 'spikes' but if they are every 800 seconds, and only a few ms long, it seemed possible. If it comes to it I will add whatever power is required but it just becomes less portable in an application I want to be as portable as possible. Again if it comes to it, there are lipos out there that are hardly bigger than a pair of CR2032s and can surely deliver a fat spike of current when required. In fact the board I am using has a very convenient lipo support built in, with direct battery connection and charging etc already dealt with. So maybe plan to go with that for now if the coins look too difficult.

Is the arduino SD library very unreliable? I didn't know. SDfat etc that is right? Whats its problems - this is even with good code behind it, not rookie stuff with poor timing and such?

14

Google it, if you don't believe me. But certainly, feel free to try, I would be happy to be proved wrong.

Is the arduino SD library very unreliable?

I gave up on it several years ago, after trying to implement data storage for greenhouse environmental measurements. I think I lost more data, through a variety of inexplicable errors, than was ever successfully saved. Other forum members have made similar comments, although I've not seen a comprehensive study.

After switching to Raspberry Pi boards, I have not once had a problem with SD data storage.

15

Does RasPi include the RP2040?

16

I don't know the current state of things, but in the past I found a good bit of variability between different brands of SD cards as to power consumption. Not all of them went to a low-current idle state at all. So you may have to do some experimenting and testing. Take a look at this:

https://neurotechhub.wustl.edu/micro-sd-card-low-power-showdown-part-ii/

17

Raspberry Pi folks designed the chip. Their Pico series implements the system on a thumb-sized board.

Designed by Raspberry Pi, RP2040 features a dual-core Arm Cortex-M0+ processor with 264kB internal RAM and support for up to 16MB of off-chip flash.

18

@ShermanP Thanks for posting the link to the idle current draw study. I was not aware that there is such a wide variation, but that is a rather limited selection of SD cards. The write currents documented for those cards also seemed to be lower on average than I've seen, but the method used to measure it may not have caught the true peaks.

The study linked below covers a wider variety of cards, and documents read/write currents exceeding 180 mA, also much higher "sleep" currents:

19

My default solution for SD idle current is not to have any at all. I typically switch the power with a mosfet. Another reason to do that is to be able to reset the card if things get locked up for some reason. The MCU could reset, but that doesn't reset the card. So a power cycle can be useful at times. You just have to allow some time after powering it up for the card's controller to initialize and be ready to communicate.

20

@anon94801460 what I meant more was do they take that much current rather than surely they don't take that much - I'm after advice on this actually, don't really have any prior knowledge (I've done a little SD reading and writing a few years ago but never with an eye on the power consumption or any other details like that). So I appreciate the heads up on that. And the experience with the monitoring system.

Re the Pi aspect, much as I do like Pi's and run several of them doing various jobs, in fact I could not live without them, I don't think any Pi is suitable for this job (or any linux running device). If I can't reliably write to an SD with the nRF52 I'll have to switch to some sort of external memory chip which is OK but I prefer not if i can help it. It would make the job simpler though.

My experience with things like the arduino based on the 328 or the stm32f104's, both of which I've used for quite a few years, is that the hardware is rock solid. Thats not to say the SD library is though of course. But the way they run, with no OS on there, works great. I do usually do 'bare metal' programming on both of those though based around interrupts which I like a lot, so I can't vouch for the arduino libraries. If there is a problem with writing SD cards on the arduino I assume its related to the SD libraries, which does create a problem for me since I want to use those this time. I don't really want to deal with the basic SD / spi protocol stuff myself if I can avoid it, it looks quite involved. That, and for the BLE libraries, is why I am looking to use the arduino system for this project.

I guess though that the lower down I go (IE the less SD library functionality I use) the less chance that I will hit any SD library bugs. So that again supports doing the most basic type of write on the cards (IE just write sectors, no filesystem etc). SD cards, and SSD's too, do all this wear levelling, and the way they do copy on write, with small writes, is a nightmare. I've lost SD's and SSD's because of it, because of power cuts during these operations I assume. So I am indeed really wary of that. I would assume that those aspects can combine with limitations of single threaded control and other aspects of the arduino to create potential points for data loss. In fact thats one thing I am looking for advice for now. As I mentioned I am wondering if writing multiples of erase block size data aligned to block boundaries should prevent the copy on write being invoked hopefully? Maybe? I dunno. And wear levelling should hopefully not come into operation on a blank device erased at the start. Again I am not sure though.

So, with the best will in the world, I can do my best and still get the type of problem you mentioned in your greenhouse system. At that point I will have to capitulate and buy some 16Gb memory chips. But I do hope it does not come to that point.