Logging 40,000 Samples/Sec to a SD!

Using Arduino Storage
1

I am finishing features in the new version of SdFat that allow data logging at higher speeds.

I posted an example plot of a waveform that was digitized at 40,000 samples/sec here Logging 40,000 Samples/Sec to a SD! - adafruit industries.

I used a12-bit MCP3201 ADC and logged the data to a file as 16-bit binary numbers using an Adafruit Data Logging Shield.

I would appreciate any feedback.

Would you use this capability?

What type data would you log?

I plan to include an example program to read files on a PC/Mac since the files are binary. What language would be best for the PC part of the example? I am considering Java, Perl, or Python.

2

fat16lib:
I plan to include an example program to read files on a PC/Mac since the files are binary. What language would be best for the PC part of the example? I am considering Java, Perl, or Python.

I'm really impressed about the performance!

Why don't you use Processing as base. I haven't used it myself, but it's portable and using Java.

3

I would appreciate any feedback.

Impressive, well done!

Would you use this capability?

Think it is usefull for a memory scope,

  • monitoring electrical signals like rs232 (until a certain baud rate)
  • recording sound - you did that before
  • visualize bouncing of a switch
  • recording pulses of a high speed axis with encoder switches on it (recording timing of an engine)

On the other hand, just the fact you can store data so fast enables you to makes continuous measurements that are almost not interfered by storing time (missing samples).

Think I would use this capability to store more data. now I often store minimum data to minimize the time.
Example: I do a analogRead() 10 times and use the average in my math and I store the average and the result.
Now I can store the individual measurements and redo the math on my PC with individual measurements or average by 3 or 5
It is a new degree of freedom

What type data would you log?

  • In a project I did an analogRead() 10 times and use the average in my math and I stored the average and the result.
    Now I could store all the individual measurements in the same time and redo the math on my PC with individual measurements or average by 3 or 5 iso 10
    // in general raw data iso cooked

  • Current project involve temperature and humidity, they do not change that often, and sensor timing is not high
    idea is to make XY plot
    // in general multi sensor readings

  • A previous project used a sensor that converted light intensity to frequency - pulses - it had no logging but with the above speed I could store quite some detail
    I'm thinking of records with { (delta) millis , counter }

question: You improved reading speed to the extreme imho. At what speed can you read from SD card?

Languages: (1) python (2) C# (3) java --- an android app?
python on place 1 as it runs on any platform, windows linux macos etc.

Rob

4

Do you have a hyperlink to your library?

I can imagine to use this as an oscilloscope with a function to save the data in real time.

UPDATE:
just found the link in an earlier thread.

5

I started my recent endeavors with posting on project guidance here:

I wanted to record decent-quality audio along with other channels of data.

Which now you've essentially entirely completed except I am using a MCP3208 for recording many channels of data.

I think a 12-bit ADC is often the minimum for recording lots of interesting sensors as it measures real physical phenomena.
My guess is that people can get away with 10-bit for things that are mechanically limited and you know "by design" the limits of your
phenomena. For discovery purposes, like using in a real chemistry/physics/biological experiment this is going to be great.

Although a 16-bit ADC which others have got working with Arduino (not sure if it can get up to 40khz tho) would be very cool now that you have such a high sample rate.

While you're at it, I would suggest storing the data in edf (European Data Format). This is what I was intending to do.
Since it is necessary to interlace the data within one file, and 2 bytes is going to be the largest sample, it seems a perfect fit.
http://www.edfplus.info/specs/edf.html

There are many free & opensource edf viewers available (including matlab). I like this one: EDFbrowser
There are even some fft graphing, convert channel to ascii, etc.

There is even a freely available edf-to-wav converter available. Nice to be able to load data in Audacity.

I know this last question is probably 'no', but can any of the wifi-enabled SD cards be used in the SD shield, and you auto-magically
get a wifi-connected device?

And I would recommend java & processing for signal analysis on the PC.

6

That EDF looks like a nice candidate for an Arduino library.

Point of attention is the (for arduino) large header of 256 + ns *128 bytes, for 2 samples this would be 512 bytes (typical sector size)
As this header is only used once it would be nice to reuse it for the array of samples - I'm thinking :

struct EDFHeader
{
...
}

struct EDFData
{
int data[][];
}

union EDF
{
EDFHeader EH;
EDFData ED;
}

just to capture the thought,
rob
-- update --
although too big for arduino (UNO) this library seems to be a good place to get started - EDFlib for C/C++ -

7

Learn how to use progmem...
Fast sampling rate is way better than having a ton of bits in the adc and then not knowing how to design low noise boards cables and connectors and how to use them in precision measurements, there are lots of scopes that use 8 bits adc's, the magic is the analog section, low noise circuitry and power supplies and good boards and cables.

If you use an 12bits adc's and them stapple and lm7805 and two caps you can kiss goodbye to your lab measures lol

8

This post is sort of a progress report and more detail on design decisions.

I am not sure what to make of Senso's post. I am not trying to build a 200 MHz scope using progmem and an 8-bit flash ADC.

It is true that if you ignore basics of low noise design with an ADC like the MCP3201 you may reduce the ENB, Effective Number of Bits, to around 5.5. See this design note from Microchip http://ww1.microchip.com/downloads/en/DeviceDoc/adn007.pdf.

I am not assuming an ADC is the source of data and am not dealing with analog design.

I am using MCP320X and MCP330X ADCs as a data source during development. I went to Digi-Key and searched for an ADC that sampled at 100 ksps or greater, serial SPI, DIP or SOIC, and reasonable price. The result was MCP300X, MCP320X, and MCP330X. I didn't limit the search to Microchip.

The MCP300X is a 10-bit ADC at 200 ksps max but since I must read the ADC with bit-bang SPI in an ISR and I can't reach 200 ksps, I decided to go with the 12 and 13 bit 100 ksps ADCs.

I have highly optimized inline bit-bang read functions for the MCP3201, MCP3202, MCP3204/8, and the MCP3301. These functions allow any digital pins to be used, three pins for the MCP3201 and four pins for the rest. The pin numbers must be constants so the compiler can optimize to simple CBI, SBI, SBIC, and SBIS I/O instructions.

The ISR is key to reaching 40,000 samples per second. Currently the ISR take about 16 microseconds per sample. This is 3.5 usec for the ISR prologue/epilogue, 10 usec to read the ADC, 2.5 usec for other stuff.

That leaves about nine microseconds per sample to write to the SD. I store 255 samples in a 512 byte block so that is a total CPU time of about 2000 microseconds to write a block.

You could slow the sample rate and read more data per sample. For example with a MCP3204 you could sample at 10,000 samples per second with four 16-bit values per sample.

Choice of SD is important. SD controllers are very different in how they handle flash. Cheap ones don't multitask. The SD spec allows a max write latency of 250 ms so cheap cards just pause for 100-200 ms occasionally to erase 128KB or 256 blocks. These just won't work in this application.

I have written a SD verify program to screen SD cards. I have found that SanDisk 4GB Ultra II cards have a consistent write time of less than 1000 usec per block. It looks like you could record a 4GB file, the max size for FAT files, without dropping a data point. That is over 13 hours at 40,000 16-bits samples per second.

I think I will use Python for my PC/Mac example programs.

I hope to release a beta of this stuff soon.

I really appreciate everyone's input. It is very helpful.

9

BTW, great work in getting this done. I look forward to seeing the code as I'm learning a lot by reading your code. Very well commented.

The TI ads8341 does 16-bit ADC at 100ksps over SPI.

some people are using it:
http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1275676171

10

wow fat, looks impressive! As soon as i get a new Micro SD card reader (dropped mine last week -.-) i'll give your code a try ... sadly that's also the reason, why i couldn't test my interrupt driven recorder ... but just hearing about your solution sounds a million times better, then using interrupts to capture data during the write process.

I'll keep a close watch on your work and will try it out, as soon as you have a working prototype :slight_smile:
oh - and i'd vote for JAVA as your parser code, as i'm coding my desktop pendant in Java as well ... to parse the byte Log, i had to use a simple fileInputStream, read the individual bytes and put them in a buffer of size 16, because when reading complete lines, the possibility of a byte having the value 13 (newline) was too high and i had incomplete lines.. If you like, I can give you my java parser, to check out what i did :slight_smile:

keep up the good work!

11

The TI device didn't pass the DIP/SOIC test. I didn't want people to face the 0.025 inch pin spacing of the SSOP-16 package.

The first sample binary read program for the PC/Mac will be super simple. It will just be a demo of how to read the file and write the values as text. I want to get this stuff out so I can get feedback.

12

looking for this?

something like

f = file('binaryFile', 'rb')
while 1:
    h = f.read(1)
    l = f.read(1)
    if not h:
        break
    print ord(h)*256 + ord(l)

Rob