Fast data logger for multiple analog pins

I have implemented a data logger to test several new features of SdFat and a small RTOS.

This logger produces a CSV file and performs very well for a logger that formats text. I have logged two analog pins at 1000 Hz on a Uno and 16 pins at 100 Hz on a Mega.

It is designed to log any number of analog pins starting with pin zero. This could be modified easily.

It works best with a quality SD card but will provide good results with lower quality cards.

Several features of SdFat and the RTOS enhance performance.

Two threads and a large buffer are used to overcome the occasional large write latency of SD cards.

A special version of analogRead() is used that sleeps during the ADC conversion. This saves CPU time for the SD write thread.

A new SdFat function, printField(), is used to format data. This function is several times faster than the standard Print class.

The logger is the nilSdLogger example in NilRTOS.

NilRTOS20130214.zip Google Code Archive - Long-term storage for Google Code Project Hosting..

Three parameters configure the program. Here is an example to log the 16 Mega analog pins at 100 Hz.

// Time between points in microseconds.
// Maximum value is 4,194,304 (2^22) usec.
const uint32_t PERIOD_USEC = 10000;

// Number of ADC channels to log
const uint8_t NADC = 16;

// FIFO buffer size. Adjust so unused idle thread stack is about 100 bytes.
const size_t FIFO_SIZE_BYTES = 4000;

Here is the beginning of the data file:

PERIOD_USEC,10000
ADC0,ADC1,ADC2,ADC3,ADC4,ADC5,ADC6,ADC7,ADC8,ADC9,ADC10,ADC11,ADC12,ADC13,ADC14,ADC15,Overruns
528,466,429,402,388,372,362,352,370,356,361,350,339,346,341,342,0
407,414,409,398,389,376,365,352,352,347,349,344,335,337,334,337,0
365,380,387,386,383,376,368,354,345,342,343,340,333,333,330,332,0
345,355,364,368,371,369,366,352,343,341,342,340,335,335,333,335,0
329,333,339,343,347,348,348,338,328,326,326,325,321,321,318,322,0
329,330,333,337,341,343,344,335,328,328,328,327,324,324,321,321,0
315,315,316,318,321,324,325,320,315,315,315,315,313,313,313,313,0
311,310,311,311,313,315,316,311,307,307,307,307,305,305,304,303,0
305,304,305,306,307,308,310,305,303,304,305,305,304,305,304,302,0
292,291,291,291,292,292,293,290,288,288,288,288,287,288,288,286,0

Statistics are printed at the end of a run. Here are stats for the Mega.

Max Write Latency: 68780 usec
Unused Stack: 49 3207
FIFO record count: 125
Minimum free count: 117

A very good SD card was used so the max write latency was about 69 ms.

The data read thread had 49 bytes of unused stack.

The SD write thread had 3207 unused bytes so more buffering could have been used. Adjust FIFO_SIZE_BYTES so the write thread has about 100 unused bytes for best buffer performance.

125 buffers were allocated with space for 16 ADC value in each buffer. The total buffer space was 4000 = 125*32 bytes.

The minimum free count was 117 buffers so there was never danger of a data overrun.

can you tell the chances this would also work on an Arduino with a 328p ?
Thanks, Robert

Yes it works on 328P Arduinos. It has been used on 1284P in addition to Mega.

I have logged two or three analog pins at 1000 Hz on a Uno and all six pins at slower rates. The maximum rate depends on your SD card.

All SD cards have occasional long write latencies. Externally, SD cards appear to have simple 512 byte write blocks.

Internally things are much more complex. Physical blocks are much larger than 512 bytes and physical blocks are mapped to logical block addresses for flash wear leveling. Flash must be erased before it can be written. This internal housekeeping can result in occasional long write times.

The SD spec allows write delays as long as 250 ms. Most consumer cards have maximum write latencies around 100 ms.

You can run the SdFat bench example to get an idea of your card's write latency. Here is an example from an 8 GB card.

Free RAM: 1027
Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test. Please wait up to a minute
Write 192.05 KB/sec
Maximum latency: 88288 usec, Minimum Latency: 88 usec, Avg Latency: 515 usec

The example is great-- because it obviously works fast. I would like to experiment with it, but I have no idea how to connect an Arduino Due with an Sd card shield (I have a SEEED Studio SD card shield v 3.1). Could you please help me out with some details?

Kindest Regards,
MJM

I have no idea how to connect an Arduino Due

Sorry, I meant an Uno and have edited the original post. It should work on your board.

It works on AVR Arduinos, 328, Mega, Leonardo, and 1284 boards.

2840? Did you intend 1284 there?

2840? Did you intend 1284 there?

Yes. I will never get this post right.

The post discussion is not that important, it's the logger code that matters :wink:

Hi-

I like this development effort, good work! I understand it too, multithreading and such is part of my day job, so no big deal there. However I am still new to Arduino microcontrollers, so I am having a little difficulty understanding how to implement this in real ife. I'd like to offer what I think you'd do and then have the experts tell me how wrong I am. :smiley: If that's ok...

In my project I will want to datalog and I will be sampling an analog input (only one at the present time). The 1000 samples/sec is more than I need, so this "fast datalogger" was/is what I was looking for to fill this need of the project. So the plan included:

  1. Getting the Adafruit datalogger shield and installing it on my Uno
  2. Incorporating fat16lib's libraries and headers/sources into the build (presuming that is how they are used)
  3. Utilizing this functionality as described in his documentation

Does that sound about right? A few questions abound:

  1. This uses I2C as I think I remember reading in the documentation. I read somewhere else that "I2C is slow" ... if true, does that affect the overall ability of this datalogger hardware and software to keep pace with a 1000 samples/sec data rate?
  2. I remember reading that 2G was about the max storage capability that these datalogger hardware devices were meant to support. Is that correct? It seems that someone posted that they were up-and-working with a 16G SD card and someone else was surprised that it was working. If 2G isn't the number, is there such a limit?
  3. It almost seems like (and please remember that I am new) the SD card in such a scenario is being used as a harddrive to store log files for later retrieval. Is this customary in the Arduino microcontroller world?

I ask that last question (#3) because I have a very real need to log the data and then provide the ability for the user to retrieve it later on. Seems easy enough, just remove the SD card, and pull it off on the PC, right? However these components of mine will be encased in a plastic enclosure and I'd really prefer that the users didn't open it to remove the SD card to get their data. So how could I allow them to access it? Maybe by Wi-fi or Bluetooth? Would there be any way to allow them to connect by a USB port and get their data, kinda like they would via a flashdrive/thumbstick/whatever-you-want-to-refer-to-it-as?

Many, many thanks!

Jerry

This uses I2C as I think I remember reading in the documentation.

No, I2C is not used. The logger reads from the Arduino's internal ADC and write to the SD using the SPI bus.

I remember reading that 2G was about the max storage capability that these datalogger hardware devices were meant to support.

SdFat supports any SD card formatted FAT16 or FAT32. SDHC cards can be as large as 32 GB. The maximum size of a file on a FAT32 volume is four GB so you must create multiple log files.

SdFat can write to SDXC cards that have been formatted FAT32. many vendors sell 128 GB SDXC cards. FAT32 is not the standard format for SDXC cards but the SdFat formatting sketch, SdFormatter.ino, will format SDXC cards as FAT32 volumes and Windows, Macs, and Linux accept this format.

fat16lib:
No, I2C is not used. The logger reads from the Arduino's internal ADC and write to the SD using the SPI bus.

Ok thanks, I couldn't remember if it was I2C or SPI. Out of curiosity, you could use both I2C and SPI simultaneously (but to do different things) in your application couldn't you? I wouldn't see why not, but I'm new, so...

fat16lib:
SdFat supports any SD card formatted FAT16 or FAT32. SDHC cards can be as large as 32 GB. The maximum size of a file on a FAT32 volume is four GB so you must create multiple log files.

SdFat can write to SDXC cards that have been formatted FAT32. many vendors sell 128 GB SDXC cards. FAT32 is not the standard format for SDXC cards but the SdFat formatting sketch, SdFormatter.ino, will format SDXC cards as FAT32 volumes and Windows, Macs, and Linux accept this format.

Good to know, thanks. So how do folk who implement something like this typically retrieve their data? Is it simply by removing the SD card and getting it directly from that? It seems like I might need to get creative with this one.

Thanks again fat16lib for your work.

fat16lib-

Allow me to ask another way. Say you store your data to an SD card and you want to enable the user to get the data without removing the SD card from your device. If you had a USB host breakout, how could you move the files from the SD card to the USB host such that a user could attach a flash drive/memory stick or their computer to the USB port and retrieve the files?

Thanks!

Why not locate the SD socket so the user can change the SD easily.

Connecting a USB flash drive to an Arduino and transferring files to it really hard. There is no point plugging a USB drive into the logger and transferring files from the SD to a flash drive. If you could write the USB drive just log directly to it.

I don't know of a clean way to move files from the SD to a PC/Mac over USB.

The ideal way would be for the SD on the Arduino to appear as an USB mass storage device. This is also very difficult.

Paul Stoffregen has spent a lot of time on this.

The best I have done is send files over a USB serial connection or have the Arduino act as a web server using a SD/WiFi shield.

One SdFat user was trying to use an Eye-Fi card http://www.eye.fi/ I don't know if the user got it to work.

fat16lib:
Connecting a USB flash drive to an Arduino and transferring files to it really hard. There is no point plugging a USB drive into the logger and transferring files from the SD to a flash drive. If you could write the USB drive just log directly to it.

Is there a way to log data to a USB flash drive with an Arduino? I don't remember seeing one. And your library uses the SD library to log data, so if there was a way to write to a USB drive, how would you log data to it? That is, what function calls would you make? Is there a library for all this stuff that I haven't found yet?

I'll look into the other options you mentioned, thanks.

Is there a way to log data to a USB flash drive with an Arduino?

Not that I know about. You would need a USB shield and a USB flash drive library.

1 Like

Hello fat16lib,

Many thanks for the file. Seems to work really well. I was trying to make this fast analog reading with the analoglogger from the sdfat library, which was having quite trouble to make it below 20ms.

Let me make a suggestion for improvement, a code that reads the already present csv files on the SDcard and doesn't overwrite them!

Keep up the good work,

Movykappa,

Thanks for the suggested improvement. Many people have suggested other features and "improvements".

The purpose of this examples is to provide a simple base for fast data logging that can easily be modified so I have not complicated the example with the many suggested additions.

I will likely add a function to SdFat to create a new file with rotation numbers.

The function would be used like this:

  if (!file.createUnique("LOG.CSV")) {
    Serial.println("createUnique failed");
    while (1);
  }

This would create and open a new empty file with name "LOGnn.CSV" where  nn is 00 - 99.  This would prevent writing over existing files.

Hello everybody.

I decided to create another analog datalogger for my RC aircraft but I need some help and tips for design.
Since I started learning FreeRTOS for Arduino, I realized soon, that the best way is to use an real time application for savings data on a SD card.

So here the general specifies:

  • Task 1 (“T1”, high priority) reads the 6 analog pins every 4 millisecs and save them in a struct. Then move the struct into a queue;
  • Task 2 (“T2”, lower priority) reads the struct from the queue and save it into a SD Card. This Task runs everytime T1 is blocked;

The biggest problem is that I don t know how to efficiently copy data from the queue to the SD card. Since it could last milliseconds, I thought 2 strategies:

  • to append during T2 the whole queue into the file and then close it => super Problem: it lasts more than 4 - 0.6 = 3.4 milliseconds;
  • to append during T2 just few values from the queue to a static buffer (which can store about 2000-3000 values) and only when the buffer is completely full then transfer all the buffer to the file and close it (during this operation I would stop T1, since I don’t need more values from the sensors).

I have no idea, how it is better to implement it. Since I don t have too much time between T1 und T2 I m afraid I can mess all data in the queue. Do you have a much better idea !??!?! Please…you are welcome!!!

PS: I know that there are more efficient RTOS out there, like NilRTOS for istance, but please, I m learning now FreeRTOS and I bought the application guide for microcontrollers, so I don t want to get crazy learning another RTOS.

Hello

Ok. I've read the documentation (the examples) of NilRTOS and I understood the idea, to create a buffer, in wich data are going to be stored and read according to the FIFO rule. And both tasks are going to be synchronized to put and read data from the buffer.
maybe it is a better idea, dont know... may I ask, if my idea to put data in the queue with FreeRTOS functions synchronizing the tasks is better than to create a single buffer like the example in NilRTOS!?!?!??

I would like to understand more....

Thank you very much

Regards

Hello,

I tryd to use this sketch for logging with a manual on/of switch on a digital pin instead of serial character.
I tryed simply to replace " while (!Serial.available()) { " with " while (!Pinbutton high()) { "
but couldn,t get it to work.
any suggestion for this Where to place the digitalpin read ?

klass