SD logger - with several sensors - not fast enough

hello,
I am using multiple sensors installed on several arduinos to log data on an SD-card.
the sensors i am using are an ultrasonic sensor, an accelerometer sensor, an OBD-II, and finally a GPS module.
the ultrasonic US-100 is sampling at approximately 90 Hz and is digitally connected on an arduino uno
the accelerometer ADXL-345 is sampling at approximately 390 Hz and is connected to a separate arduino uno via i2c connection
the OBD-II UART sparkfun along with the arduino GPS shield are sampling at 1 Hz and are connected serially to the same arduino mega

all above arduino boards are connected in turn serially to an arduino mega as master-slave, after which i intend to log the received data onto an SD card connected via SPI unto that master arduino mega.
any idea how might i be able to log the data acquired as fast as possible without any issues directly onto the SD memory card?

thank you in advance

*OBD-II details ==> https://www.sparkfun.com/products/9555

*US-100 details ==> http://www.e-gizmo.com/KIT/images/ultrasonicsonar/ultrasonic%20sonar%20module%201r0.pdf

*GPS shield details ==> http://www.electronicaestudio.com/docs/ISTD-030hd.pdf

*Accelerometer ADXL-345 details ==> SparkFun Triple Axis Accelerometer - ADXL345 - Solarbotics Ltd.

*SD card logger details ==> http://robotbase.en.alibaba.com/product/367642250-210608919/SD_Read_Write_Memory_Module_SD_Card_V2_0.html

I've also been playing around with a data acquisition system and have been tackling the same issues.

There are two main limitations that I have come across:

  1. Serial transmission is pretty slow. Try to minimise the amount of stuff you send over serial, use the fastest stable clock rate, and where possible use faster communications standards, i.e. move to the right you can and if speed is a factor: USART >> I2C >> SPI.

  2. RAM is limited so you will have to transmit data sufficiently frequently so that you don't run out of buffer.

In my own case, I found that on an Arduino Uno I could theoretically sample three analogue and five digital channels at 500Hz if all I was doing was reading ADCs and saving to a memory buffer. On an Arduino Due I could do the same thing at 5kHz. If I started then transmitting that data over USART to a computer then the sample rate RAPIDLY flattened out:

If you follow the blue line you can see that as I ask the Arduino to sample faster and faster the average sample rate actually achieved flattens out. When I tell the board to sample at its maximum 500Hz it actually only achieves an average rate of about 75Hz. In reality what it is doing i sampling in very short bursts then spending about four times longer ignoring the sensors and sending data via serial. In other words, you may have about 1 second @ 500Hz then 4 seconds of nothing, then 1 second @ 500Hz and so on.

Your data format may differ slightly, and may be a bit more efficient as I'm required to store a fair bit of meta data such as timestamps, digital flags, board IDs, etc. You could also look at where using a faster board such as a Due could improve bottlenecks (your "master Mega" seems to be doing a lot of serial comms) or if you could consolidate some of the sensors onto less Arduino's and somehow reduce serial traffic this way. I'm guessing the main issue is going to be the accelerometer!

Good luck!

It is possible to log much faster.

See this Try this super fast analog pin logger - Storage - Arduino Forum.

Here is a table of maximum sample rate vs number of analog pins in a sample. A Mega was used for more than six pins.

			ADC clock kHz		
	125	250	500	1000	
pins					
1	7692	14286	25000	40000	
2	3810	6667	11111	16667	
3	2572	4790	8421	13559	
4	1942	3636	6452	10526	
5	1559	2930	5229	8602	
6	1303	2454	4396	7273	
7	1119	2111	3791	6299	
8	980	1852	3333	5556	
9	872	1649	2974	4969	
10	786	1487	2685	4494	
11	715	1354	2446	4103	
12	656	1242	2247	3774	
13	606	1148	2078	3493	
14	563	1067	1932	3252	
15	525	996	1806	3042	
16	493	935	1695	2857

A RTOS is more general solution. See Google Code Archive - Long-term storage for Google Code Project Hosting.. Look at the NilRTOS nilSdLogger example.

fat16lib:
It is possible to log much faster

Interesting data.

Obviously the logging rate is dependent on a lot of things. i.e. what do you do with the data? do you take a number of samples and average for one reading? what sort of meta data do you capture? how much RAM do you use for storage?, etc. etc. My example was simply to show that the limiting factor was the serial comms, not the capture rate, which your numbers support. :slight_smile:

what do you do with the data?

I write it to the SD card.

do you take a number of samples and average for one reading?

No I write all samples to the SD card.

how much RAM do you use for storage?

RAM is only used for temporary buffering.

My example was simply to show that the limiting factor was the serial comms,

You can log data at much higher rates over serial than you are achieving.

what sort of meta data do you capture?

Metadata is a fairly ambiguous term if you go beyond the definition "Metadata is data about other data". What do you mean by this statement?

@16fatlib:
I think that few newbies know your forum reputation... I know you are generally modest but you could probably quelch a lots of dialog by introducing yourself as the author of SD library XD

Assuming you are not just toying with your prey. Yo tha man, dude.

Ray