SD logger - with several sensors - not fast enough

hello, I am using multiple sensors installed on several arduinos to log data on an SD-card. the sensors i am using are an ultrasonic sensor, an accelerometer sensor, an OBD-II, and finally a GPS module. the ultrasonic US-100 is sampling at approximately 90 Hz and is digitally connected on an arduino uno the accelerometer ADXL-345 is sampling at approximately 390 Hz and is connected to a separate arduino uno via i2c connection the OBD-II UART sparkfun along with the arduino GPS shield are sampling at 1 Hz and are connected serially to the same arduino mega

all above arduino boards are connected in turn serially to an arduino mega as master-slave, after which i intend to log the received data onto an SD card connected via SPI unto that master arduino mega. any idea how might i be able to log the data acquired as fast as possible without any issues directly onto the SD memory card?

thank you in advance

*OBD-II details ==> https://www.sparkfun.com/products/9555

*US-100 details ==> http://www.e-gizmo.com/KIT/images/ultrasonicsonar/ultrasonic%20sonar%20module%201r0.pdf

*GPS shield details ==> http://www.electronicaestudio.com/docs/ISTD-030hd.pdf

*Accelerometer ADXL-345 details ==> https://solarbotics.com/product/50656/

*SD card logger details ==> http://robotbase.en.alibaba.com/product/367642250-210608919/SD_Read_Write_Memory_Module_SD_Card_V2_0.html

I've also been playing around with a data acquisition system and have been tackling the same issues.

There are two main limitations that I have come across:

1) Serial transmission is pretty slow. Try to minimise the amount of stuff you send over serial, use the fastest stable clock rate, and where possible use faster communications standards, i.e. move to the right you can and if speed is a factor: USART >> I2C >> SPI.

2) RAM is limited so you will have to transmit data sufficiently frequently so that you don't run out of buffer.

In my own case, I found that on an Arduino Uno I could theoretically sample three analogue and five digital channels at 500Hz if all I was doing was reading ADCs and saving to a memory buffer. On an Arduino Due I could do the same thing at 5kHz. If I started then transmitting that data over USART to a computer then the sample rate RAPIDLY flattened out:

If you follow the blue line you can see that as I ask the Arduino to sample faster and faster the average sample rate actually achieved flattens out. When I tell the board to sample at its maximum 500Hz it actually only achieves an average rate of about 75Hz. In reality what it is doing i sampling in very short bursts then spending about four times longer ignoring the sensors and sending data via serial. In other words, you may have about 1 second @ 500Hz then 4 seconds of nothing, then 1 second @ 500Hz and so on.

Your data format may differ slightly, and may be a bit more efficient as I'm required to store a fair bit of meta data such as timestamps, digital flags, board IDs, etc. You could also look at where using a faster board such as a Due could improve bottlenecks (your "master Mega" seems to be doing a lot of serial comms) or if you could consolidate some of the sensors onto less Arduino's and somehow reduce serial traffic this way. I'm guessing the main issue is going to be the accelerometer!

Good luck!

It is possible to log much faster.

See this http://forum.arduino.cc/index.php?topic=228549.msg1650827#msg1650827.

Here is a table of maximum sample rate vs number of analog pins in a sample. A Mega was used for more than six pins.

            ADC clock kHz       
    125 250 500 1000    
pins                    
1   7692    14286   25000   40000   
2   3810    6667    11111   16667   
3   2572    4790    8421    13559   
4   1942    3636    6452    10526   
5   1559    2930    5229    8602    
6   1303    2454    4396    7273    
7   1119    2111    3791    6299    
8   980 1852    3333    5556    
9   872 1649    2974    4969    
10  786 1487    2685    4494    
11  715 1354    2446    4103    
12  656 1242    2247    3774    
13  606 1148    2078    3493    
14  563 1067    1932    3252    
15  525 996 1806    3042    
16  493 935 1695    2857

A RTOS is more general solution. See https://code.google.com/p/rtoslibs/. Look at the NilRTOS nilSdLogger example.

fat16lib: It is possible to log much faster

Interesting data.

Obviously the logging rate is dependent on a lot of things. i.e. what do you do with the data? do you take a number of samples and average for one reading? what sort of meta data do you capture? how much RAM do you use for storage?, etc. etc. My example was simply to show that the limiting factor was the serial comms, not the capture rate, which your numbers support. :)

what do you do with the data?

I write it to the SD card.

do you take a number of samples and average for one reading?

No I write all samples to the SD card.

how much RAM do you use for storage?

RAM is only used for temporary buffering.

My example was simply to show that the limiting factor was the serial comms,

You can log data at much higher rates over serial than you are achieving.

what sort of meta data do you capture?

Metadata is a fairly ambiguous term if you go beyond the definition "Metadata is data about other data". What do you mean by this statement?

@16fatlib: I think that few newbies know your forum reputation... I know you are generally modest but you could probably quelch a lots of dialog by introducing yourself as the author of SD library XD

Assuming you are not just toying with your prey. Yo tha man, dude.

Ray