I'm working on a project to log car oil pressure, rpm, temperature etc to an SD card. I'm using an ATmega1284P I bootloaded and fat16lib's great SdFat library and AnalogLogger example. It's working pretty well with no edits from me, logging six analog pins at approximately 20ms. This is faster than I need, the maximum sampling rate I need is for oil pressure (say every 100ms). The other metrics like exhaust and water temperature can be slower e.g. every second to save SD space.
Some of my sensors are digital e.g. DS18B20 water temp sensors and my RPM sensor which is a hall sensor counting the pulses from a toothed wheel which I am measuring using the pulseIn function. I have done a bit of basic arduino programming but the AnalogLogger code is way beyond what I have done before - it looks like more C++ coding than the basic arduino functions I have used in the past. I'll need to edit it to include the non-analog sensors.
I've read through the code and get a sense of it. Data from analogRead is written to a buffer and then flushed out periodically but I don't really understand exactly how it works e.g. ofstream and bout << ',' << analogRead(ia);. I', hoping I can edit the extract below to include my DS18B20 and hall sensor reading.
// read analog pins and format data
for (uint8_t ia = 0; ia < SENSOR_COUNT; ia++) { #if ADC_DELAY
analogRead(ia);
delay(ADC_DELAY); #endif // ADC_DELAY
bout << ',' << analogRead(ia);
}
bout << endl;
// log data and flush to SD
logfile << buf << flush;
Would really appreciate any advice / ideas - should I attempt to edit the code above and stick with the SdFat example (which is probably beyond me!) or go to the other SD library example.
It's working pretty well with no edits from me, logging six analog pins at approximately 20ms.
I suspect you will lose data points occasionally. SD cards have occasional write delays of 100 or more ms. These delays happen when the SD controller does large flash erase operations or moves data for flash wear leveling. The SD specifications allows write delays of up to 250 ms. Most modern cards have shorter delays but few cards are capable of logging data reliably at 20 ms.
Have you checked the interval between points?
If missed data points can't be tolerated, you will need to use a more complex logger based on an RTOS or collecting data in a timer driven interrupt routine.
Thanks for your replies, I will do some more reading and get used to the coding nomenclature - thanks for the pointers John. I'll try and insert a simple piece of code like digitalRead to record whether a pin is high or low and writing it to the SD every second as opposed to ten times each second for oil pressure if that is possible (to minimise storage space required).
Thanks fat16lib, it would be better if I could ensure that I didn't miss points. I did a four minute test run sampling as quickly as possible (LOG_INTERVAL set to 1) and had only thirteen data points where there was a gap of more than 100ms between consecutive points. These seem to have occurred at 67-68 seconds and 158-168 seconds. The maximum lag between points was 420 milliseconds. It seems like there are possibly two events causing the delay (one lasting about 100 - 160ms and one lasting approx 420ms), though maybe I am reading too much into this short experiment! Sorry for the poor table format, I am new to forums...
Unfortunately "high end" cards like your SanDIsk HC I don't perform well with Arduino. Arduino uses SPI mode and has little buffering so it uses single block commands.
These cards were designed for multi-block streaming in 4-bit SD mode with very large writes to flash. This results in poor performance with Arduino.
With 1284p you may log 30kB/sec under Nilrtos without loosing single byte. Afaik it has no sense to mess with special sdcard types for a datalogger - instead a robust sw solution is needed - therefore I recommend you the Nilrtos with its FIFO implementation.
Make a task which will scan all your sensors each XXmsecs and it will fill in the FIFO with sensor's data. Make the second task, which will check whether the sdcard is ready and it will then empty the FIFO accordingly. The FIFO buffer few kBytes large will eliminate the sdcard's outages.
The FIFO size to estimate is easy:
FIFO size = max_sdcard_latency * data_rate
For example:
max_sdcard_latency = 250msec
data_rate = 400 bytes to log each 20msecs
FIFO size = 250msec * (400bytes/20msec) = 5kBytes
As fat16lib said the sdcards have large writing latencies, moreover of an unpredictable nature.
Thank you both that's great, looks perfect - I'll have a play around with that.
One thing I can't work out at the moment... I can run the nilSdLogger sketch no problem on my Uno with Deek Robot SD shield, but when I try on my 1284p (which I have stripboarded - I've attached a photo) I get an error:
SD problem
SD errorCode: 0X4,0XFF
The weird thing is that the card works fine in my 1284p with the other SD related sketches I have run, but not seemingly with this one at the moment.... I will have a play with the settings, I have tried changing the chip select from SS to 4 which I'm pretty sure is consistent. Incidentally I am using 1284p bootloaded with "Mighty 1284p 16MHz using Optiboot" (Maniacboot) and IDE 1.0.5-r2, I am assuming these are compatible with the nilSdLogger.
Just to mention, I get no serial prompt with any sketches so if for example with SDInfo you are normally prompted to type any character to start after opening the serial monitor. I never get the prompt with my 1284p but it still works if I type a character, it then returns details of the card in the serial monitor.
Not sure if this is connected...I've had a quick play with the settings and can't seem to get it going at the moment.
The sdcard uses 4,5,6,7 (pins 5,6,7,8) on the 1284p.
It seems you are using CD4050 there as the level shifter- that is not the chip you have to use - you must use 74HC4050 instead.
The old CD4050 will most probably not work (not rated for 3.3V and slow).
The lost serial data - two potential issues there:
CD4050 most probably does not work with 3.3V and that serial speeds properly - and you use it as the level shifter for TX, and/or,
also mind some HC-05 firmwares have got a timeout of 5secs when TX is idle, then it goes to a sleep. Waking up the HC-05 from the arduino TX side takes some time, so your incoming data may get lost.
PS: You may consider single 3.3V power. The 1284p works fine @16MHz and 3.3V. Your life gets much easier with single 3.3V, no messing with level shifters, also a single lion/lipo cell with an LDO regulator does the job then..
Hi Pito, you have sharp eyes to get all that from my grainy photo! Thanks that is really helpful, I've ordered a 74HC4050N... a poorly performing level shifter certainly would fit in with the symptoms. As the pinout is the same versus the CD4050 I should be able to slot straight in to compare.
Thanks for the idea to go to 3.3V, I think it is a good one which I could implement without many changes. Would that affect accuracy of my analogue readings as less volts per division for the ADC? The pressure sensor I am using gives 0 - 5V output but presumably I could just run through a potential divider to scale to 3.3V?
PS I am currently monitoring via USB so the bluetooth module is not a factor I think. It is currently powered off and I am using via RX/TX1 so should not be interfering.
Would that affect accuracy of my analogue readings as less volts per division for the ADC?
No. There is on-chip 2.5V or 1.1V reference voltage for the ADC. Use that instead of 5V.
The pressure sensor I am using gives 0 - 5V output but presumably I could just run through a potential divider to scale to 3.3V?
Yes, a resistive divider will work fine.
The RTCs uses open drain (the I2C standard) so you do not need a level shifter there (or use something newer than the obsolete DS1307). There are zillion of op-amps running @ 3.3V, all intelligent I2C/SPI sensors I know run @ 3.3V, all sdcards run @3.3V, all BT modules are 3.3V, all RF modules are 3.3V, all W5100/5200 stuff is 3.3V, all SPI/I2C SRAM/MRAM/FRAM memories are 3.3V, all newer displays are 3.3V (and the older run well with 3.3V signals), etc. Even LEDs run well at 3.3V :)..
Frankly I do not understand people building their own boards still mess with 5V
3.3V definitely certainly does have a lot of advantages then - it would save me a few components which is always good! I've hooked up the 1284p to the SD card shield and everything worked as it should so I think the CD4050 is the problem with the SD problem and will be solved by the 74HC4050.
The lost serial data I think may be caused by my removing the following line of code, which won't compile when I select atmega1284 (get the error message "no match for 'operator!' in '!Serial'"):
while(!Serial) {} // wait for Leonardo
If I remove the ! operator I get error message "could not convert 'Serial' to 'bool'" I think because 'Serial' isn't included in the Optiboot serial library?
The lost serial data I think may be caused by my removing the following line of code, which won't compile when I select atmega1284 (get the error message "no match for 'operator!' in '!Serial'"):
The official Arduino boards reset when the serial monitor is opened.
One of the hardware flow control lines (DTR) of the ATmega8U2/16U2 is connected to the reset line of the ATmega328 via a 100 nanofarad capacitor.
If you do not implement this feature on your 1284P board, you will not see the first serial output.
Thanks so much for all your help, sorry for all the silly questions! Not having the dtr connected up was preventing me receiving the first serial input. I tried to modify an old usb-ttl I had lying around but damaged it soldering pin 2 via the capacitor to the reset. I'll use my uno board with the chip removed to upload sketches and monitor the serial for now (or use my bluetooth module)...
I'll have a go at bringing my code in for the digital sensors next.
Ah got you... I'm being slow! I'll have to figure out a way to sort it out without buying something new. Perhaps I can pull the reset pin low - software reset or something.
