My final year project at uni is to design a vehicle data logger for a racecar that requires several analog and digital inputs to be logged at 400+ Hz. I have decided to use a couple Arduino megas/deumi with the sparkfun uSD shield, CAN-bus shield and some thermocouple amp breakouts (SPI).
I started with some very basic programs and have slowly got something that is promising. Using the deumi and the sd shield (with Bill Greiman's newest Sdfat library) I have been able to get read speeds from 2 linear pots (0-5V analog outputs) at over 2500 Hz. Still trying to figure out how to get this to work on the mega (using software SPI instead of jumpers from pins 50-53). However, my call for help/advice here is to get some comments on my design plan to meet the remaining criteria for the vehicle DAQ.
The plan is to increase the analogreads (and the size of the buffer they are stored in) until all analog inputs are captured while using ISRs/attachinterrupts to get data from the enabled digital pins that are connected to hall effects sensors and the like (which output HI/LO pulses I can detect and convert to rates [e.g. rpm's from the hall effect] with counters and the millis/timer functions).
Is this an efficient way to approach the problem? How do I ensure the sampling times are evenly spaced? Will using ISRs as mentioned take care of this?
It is now working with the mega. Forgot to add code before: The following code runs a read test for 1 second (while loop) and prints the input values and a read speed in Hz.
//Add the SdFat Libraries
#include <SdFat.h>
#include <SdFatUtil.h>
#include <ctype.h>
//Define Pins
int pot1Pin = 8;
int pot2Pin = 9;
int start;
long n = 0;
//Create the variables to be used by SdFat Library
Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;
char name[] = "log.csv"; //Array that contains name of file.
char contents[32]; //Data buffer for writing contents to the file.
#define FASTADC 1 //CODE TO INCREASE ANALOGREAD SPEED BY CHANGING PRESCALER
// defines for setting and clearing register bits
#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif
void setup()
{
card.init(SPI_FULL_SPEED, 8); //Initialize the SD card and configure the I/O pins.
pinMode(53,OUTPUT);
volume.init(card); //Initialize a volume on the SD card.
root.openRoot(volume); //Open the root directory in the volume.
#if FASTADC
// set prescale to 16
sbi(ADCSRA,ADPS2) ;
cbi(ADCSRA,ADPS1) ;
cbi(ADCSRA,ADPS0) ;
#endif
//Create header
file.open(root, name, O_CREAT | O_APPEND | O_WRITE); //Open or create the file 'name' in 'root' for writing to the end of the file.
sprintf(contents, ", , \nPOT1, POT2");
file.println(contents); //Write the 'contents' array to the end of the file.
start = millis();
while (millis()-start <= 1000){
n++;
sprintf(contents, "%i, %i", analogRead(pot1Pin), analogRead(pot2Pin));;
file.println(contents);
}
file.println(n);
file.close();
}
void loop(){
}
Another problem I havn't got an answer to yet is when I take out the while loop test for 1 second and let it run indefinitely, how do I ensure it writes the captured data to sd (and closes the file, without which there is no file on sd at all!). Are there buffer size issues for this case as well?
Any advice/strategies and improvements and responses to any of my many questions would be greatly appreciated!!
In your project, the opening and closing the file will consume a significant portion of the cycle of opening a file. writing a record to it, and closing the file.
Buffering the data for some time, then opening the file, writing all the records, and closing the file will result in significantly more data being written per unit of time.
How often to actually open/close the file depends on how much data you write per record (i.e. how much data you must buffer for each write cycle.
You could open the file once every 10 seconds, write to it without buffering (much), and close it again. This way, the most data you would lose is that which was written after the last close/open.
You haven't said how much data is in each record, so definitive answers aren't possible.
Yeah I noticed that opening and closing seems to take a few cycles! I was able to speed things up by putting the open and close file commands outside of the loop though (otherwise the write speed was much much worse like 12Hz i think).
In terms of total amount of data it will vary depending on which module but generally they will have 8 analog inputs (integers) and up to 5 digital ins (interrupt pins using counters to detect no. of pulses - so perhaps long ints). They do not have to be logged at the same frequency but that is easier I think. And also, does buffering for longer mean more missed samples and a more uneven sampling process (i.e. you have to wait and not sample at all while you are writing to the sd card??)
They do not have to be logged at the same frequency but that is easier I think.
I do, too.
And also, does buffering for longer mean more missed samples and a more uneven sampling process (i.e. you have to wait and not sample at all while you are writing to the sd card??)
Depends on how you do the buffering. If you use a array, and straight linear access, then, no data can be added to the buffer while the data is being written out.
If you use an array, as a ring buffer, then data can be added to/overwritten in one spot in the array while writing to the SD card occurs at another spot.
The key with a ring buffer is to (be able to) write (empty) faster than you can store (fill), so the insertion point never passes the processed point.
Thanks Paul, I had a look at ring buffers . Would you store all the values from the sensors as a string then store that into one element of the ring buffer array or have the integers read from analogread each stored to one element of the array? Any good examples you know? I saw one that was pretty complex and daunting but will try to learn how it works. I guess the main problem I've run into now is that when I pull the power, nothing is written! (as the close file command is after the loop). So I think I should have two loops and write regularly to SD card.. otherwise whenever the car is turned off the data logger will die and all data since the last file close command will be lost :~.
Would you store all the values from the sensors as a string then store that into one element of the ring buffer array or have the integers read from analogread each stored to one element of the array?
Depends on the purpose of the file. If the file is to be used on a PC, it needs to be a text file. Converting the data from integer to string format takes time and more memory, so I would do that only at the time that I was writing the data to the file.
I'd have a ring buffer of type struct XXX, where the structure had members for each of the sensor's output. Advancing the pointers is just the same as using a collection of ring buffers of type int, and the data access is only marginally more difficult (x.y[n] instead of x1[n], x2[n], x3[n], ...).
If you can afford to loose any data, you could, on each pass through loop, write the next record if there is one, and see how long the file has been open. If more than x seconds, close the file, and reopen it.
Are you using an interrupt to know when it is time to sample the sensors again?
Using structures is definitely a good idea. Losing data is not ideal as the data just before a power loss is probably the most important - however I have worked around this to ensure that when the mains power goes off there is a backup supply that will be able to power the logger for a few seconds (enough to detect a power failure and close the file on sd).
At this stage I am not using interrupts but just sampling as quick as it will let me inside a loop with analogreads. I will need to use interrupts to sample pulses from my digital sensors. Not entirely sure how I go about using interrupts to sample analog channels (timer interrupts?)
Thanks, I had a look at Mstimer2 however, it didn't really work for me as it kept repeating even when I specified it to only run for one second inside a while loop. Firstly, not entirely sure it does work properly on a mega2560 (doesn't say it does anywhere although it does for other chips).
I went back to trying to figure out a simpler scheme with analogreads and digital interrupts - and a more complex buffering scheme which will hopefully make it faster. Arrays and structures are making things tricky though (tried linklists with malloc etc.). Will continue to work on it and keep updating progress..