Portable Low power 200Hz 18bits time-stamped sd card data logging over 24h : HowTo ?

Projects General Guidance
1

Hello,

this post is about both hardware and the related software (is this "hardware" section appropriate ?)

I am at the stage of determining the necessary hardware and software, that would allow a uninterrupted low power (24h or 48h portable) 12 to 18bits data logging at 200 Hz to a SD card.

The goal is to provide me with a home-made portable ecg recording system.

Looking at various sources of info with search key words such as ‘fast portable sd card data logging arduino’

It appears that the SdFat with its 512 byte buffer might be a possible solution.

I want to interface concurrently an ecg chip such as the Max30003 (spi interface), an sd card logger (spi interface), and a real-time clock (I2c interface) ; and possibly upon temporary user choice activate a 240X320 display (spi interface), to pick a view of the incoming data, without interrupting the SD card logging ;
I want a 3 volts system (and not a 5 volts one), in order to use either a lithium battery or LiFePo4 battery with a appropriate buck regulator delivering 3V to the whole system.

QUESTIONS
1 : Would an esp32 (or an arduino nano?) do the job (esp32 with wifi/bluetooth inactivated to save power) ? What would be good arduino-IDE compatible alternate choices ?

2 : is the requirement ‘without interrupting the SD card logging ‘ realistic if the display is activated ?

3 : Is the sdfat library an appropriate choice ?

4 : what about the power budget ?

4.1 : I understand from older posts (circa 2015) that the the power consumption of various flavors of SD card varies wildly ; and that I should stay away if possible from « (circa 2015) high capacity » SD card ; However looking for example at AliExpress, the lowest sd card capacity I could find is already 8Gb : what is the current situation of various power consumption with 8 Gb sd cards ? ; should I beware of the class of the SD card ?
Is there an identified current winner in term of popular SD brand/model reference) ?

4.2 : I have seen some posts where the power to the SD card is lowered (e.g. via a P-Mosfet in series with the power supply to the SD card : taking into account the requirements of data acquisition load), is it worth considering such additional hardware ?

4.3 : For the temporary display (320X240 ILI9341 that I have) , I am of course considering such power control (via a P_mosfet) for inactivationg the display consumption when the display is not used : but is the arduino library (ILI9341_t3) compatible with such supply power manipulation, for putting to sleep/awakening without problem ?

5 : is it worth and appropriate to consider adding a memory chip (spi interface) to increase the buffer size for dumping the data in larger chunks onto the SD card ?

6 : how to avoid ecg data loss when the real-time clock will be accessed (say every nearly full buffer, and complementing the data in the buffer with the date/time before dumping it to the SD card ?

7 : should I look into a real-time OS ? For the present project as described, Is it worh the pain and the steep learning curve ?

7.1: in such case (RT-OS), are there worked-out solution for the « fast » SD card data logging (speed compatible with the requirements of the present project?

Thanks,

2

Is this a project that might actually get used for real time ECG monitoring on real people ?

I have not found SD cards on Arduinos that reliable, especially when saving lots of data, especially on ESP32s and also when there are other SPI devices on the same bus as the SD card.

3

Hello srnet,

this is for ecg monitoring on me (arrythmia; and I assume the risks :slight_smile:

4

Here is the problem in simple terms. While a 512 byte block is being written to the SD card, NOTHING else can be done in your program. So you will NOT be recording any ECG information during that time.

5

Would DMA on esp32 be susceptible to do the job in the case of other SPI devices than the SD card ?
(esp-idf/examples/peripherals/spi_master/lcd/main/spi_master_example_main.c at master · espressif/esp-idf · GitHub) ?

6

Hello Paul_KD7HB,
your point is that using an RT-OS is the only option ?

7

Never tried it, I do tend to stick to standard software.

Especially on the ESP32, I would keep the SD card on its own bus. I have also found that the 3 wire MMC interface to be more reliable than SPI.

But if you design to have the SD card is on its own bus, you can freely experiement.

9

Hello Paul_KD7HB,
what about the interrupt-driven data acquisition ?

10

so the solution below might be interesting ?

11

Can you buy it ready built ?

12

Hopefully that is how you are currently doing it! Still have the time delays writing a 512 byte buffer to the SD card. You can implement double buffering, if you are clever at programming.

13

My philosophy is KISS. Keep it simple.

I don't know much about SD power requirements so I can't help there.

I would use an ESP32, mostly due to the 3.3v operation which matches the SD voltage levels.

Most/all? displays remain at whatever was displayed until you update, and you can usually update only parts of the display. I would use a library that does only alpha/numeric and no graphics.

For time logging I would initiate the program writing the full time and date. From then on I would only write the data, knowing the sample rate you know the time between data entries.

I would write the data in binary (as opposed to ASCII). This will require you to "decode" the data to read but will be much faster.

Do you need an SD card or could you get by with EEPROM(s). Or you could look into FRAM. I've not calculated out the memory you require so this is just a suggestion.

Most devices work down to 2.2 volts or so. You may find you don't need a buck converter. Trying to get 12 to 18 bits of data in a noisy environment can be a challenge . (most buck converters make a lot of electrical noise)

14

Yes, but I do not want to re-invent the wheel; From a 2014 old post,
Try this super fast analog pin logger - Using Arduino / Storage - Arduino Forum and
https://github.com/greiman/SdFat/tree/master/SdFat/examples/AnalogBinLogger
A clever programmer (Greiman [alias fat16lib]) demonstrated that using a 4 GB SanDisk card he could log a single pin of Arduino ADC at 40,000 samples per second;
"did a reliability test logging five analog pins at 5,000 samples per second. This is an ADC rate of 25,000 values per second. I logged 512 MB of data without dropping any values."

There are nevertheless several points of attention:

  • If I adapt the interrupt to read ecg data via spi (instead of acquiring ADC data), how to gracefully handle concurrent spi access
  • what about the updated status for suitable SD cards that is readily availlable as of 2024 for handling nicely such a specific task of zero data drop while saving a continuous stream of data via an arduino, with (i) low delay for writing a buffer of data (as experimented by Greiman), and (ii) low power
  • The intermittent powering down of the SD card (after a buffer write has been completed), then powering it on before writing data of a newly filled buffer): (i) is it a viable strategy , and (ii) is it worth the power saving
15

Actually, I would like to get a plot a few seconds of the currently availlable ecg data; so I need graphics

16

But then you cannot use the method of writing to the SD card that everyone else uses.

17

Actually, assuming 2 bytes /data point, and 200 Hz data acquisition rate,
this gives 2 X 200 X 3600 X 24 = about 35 Mega bytes of data over 24h.

this is too big for EEPROMS (e.g. https://www.allaboutcircuits.com/news/microchip-releases-largest-capacity-eeprom-chip-available-to-market/)
For FRAM, I had in mind that such device might be too slow, and maight be expensive/

18

Yes, thanks for raising this important point. I might be a bit too optimistic with regard to the real-life level of immunity of the Max30003 ecg chips with respect to supply voltage fluctuations (and the possibility to get away via a large capacitor or capacitor/inductor combination); nevertheless, the Max30003 chip actually works at something like 2 volts, so that there is an additional regulator on the breakout board to reduce the supply voltage from 3V to 2V.

19

I do not care whether I use a main-stream method of SD card writing, or opt for a clever one such as Greiman's SdFat: my goal is to get the job done

20

I agree with you;
( If feasible, it would be nice in principle to incorporate real-time clock data every time a current large chunk of data has been saved, and so on until the completion of the logging).
But again, along the KISS strategy, storing the real time clock onto SD once upon starting SD card data logging, and once just before closing would be fine.

22

I agree;
I expect that the conversion of ascii data to binary is not a bottleneck;
And it is better to use a binary format so that the data to be stored is more compact (hence entailing for example a lower frequency of SD card access for a fixed size buffer)