off load serial data recording to dedicated 328P and SD card

I needed a cheap implementation SD recorder for a 9600 BAUD stream from my Mega2560. The Mega is running dual SPI instrumentation amplifiers for exhaust gas thermocouples, I2C for realtime clock, and bunches of analog inputs for fuel level, battery current, cabin temperature, ambient air temperature, 2 ports for cylinder head temps, and the list goes on. In addition to acquiring data, the Mega is managing two 16x2 line OLED serial displays. Adding SD card log recording to this code seemed a bit foolish. And so I went looking for alternatives and found them starting at $40 and running to $70 for generic recording. I was not impressed.

The SD card breakout board the plane owner purchased was the LC Studio card, discussed in the forum elsewhere and in a somewhat dim light. But I had it, so I figured I at least needed to try it before telling my friend to shell out cash for the AdaFruit or SparkFun it alternative shield plus more for an UNO to mount the shield.

I found the schematic for the LC Studio board from a link in this forum. The 3.3V linear regulator can manage 700mA plus. The idea of a naked ATmega328P-PU being back fed from the 3.3V regulator on the SD card breakout began to develop. No SPI level shifting! Dealing with the incoming serial data from the 2560 is easily done with 2 resistors.

I started the prototype and utilized the <SD.h> library in Arduino 1.0.1. The final design implements a program buffer if 512 bytes and the buffer is purged when full or when a CR is received in the stream. The CR implementation was programmed to minimize any data line losses from my logging implementation in the 2560.

The final beta code will log a substained serial stream at 9600 BAUD without character loss. The unit may go higher, but I met my design goal and my FOX character generator is switchable from 4800 and 9600 only.

The beta code, pictures, and write-up are published below. Remember, this circuit is running at 3.3V, so you will need to use level conversion for the serial input line, or a zener, or a resistor bridge to reduce the TTL signal to a 3Volt swing. There are many examples of how to do this on the Internet, but a good discussion is also here:
https://learn.sparkfun.com/tutorials/logic-levels/all

Ray

Pix3.jpg

Pix4.jpg

Datalogger3.zip (3.21 KB)

Any chance you could post larger copies of those 2nd and 3rd images?

The thought of using 2 x mcu in a project appeals greatly. I think the most recent Gallery post had a larger than life robot using 7...

I personally would have wrote interrupt-driven UART logger what you can do is in your main loop check if a buffer reaches a certain threshold and if so write the bytes to the sd card just be sure not to use SD.h a second microcontroller should be a last resort.

I wrote a Serial Logger that is based on a Serial library I wrote and my SdFat library.

It is in this download SerialPort20130222.zip Google Code Archive - Long-term storage for Google Code Project Hosting..

Here is the readme file:

This is a demo of a data logging sketch that is capable of logging serial
data to an SD card at up to 115200 baud.

Normally the maximum rate for a 328 Arduino is 57600 baud.

It is possible to log data at 115200 baud with a 328 Arduino if you use a high
quality SD card. I used a 4 GB SanDisk Extreme HD video card(30 MB/s version).

Lower quality cards can have an occasional write latency of over 100
milliseconds. This will cause a receive data overrun at 115200 baud since
the RX buffer on a 328 Arduino is 1024 bytes. You need to reduce the baud
rate to 57600 if your SD card causes receive overruns.

Most SD cards will work with a Mega Arduino at 115200 baud since the RX
buffer is set to 4096 bytes.

It is possible to use software SPI on a Mega by setting MEGA_SOFT_SPI nonzero
in SdFatConfig.h. This will allow a shield like the Adafruit Data Logging
shield to be used on a Mega without jumper wires.

This folder contains two libraries:

A beta version of SerialPort, a new library for Arduino USART serial ports.
This library allows the RX buffer size to be set in the SerialPort object.

A beta version of SdFat that allows uses of SerialPort with SdFat.

Extract these libraries from their zip files and install them.

The two programs, SerialPortLogger.pde and SerialDataSource.pde, demonstrate
high speed logging of serial data.

To run this demo, you need two Arduino boards and an SD shield or SD module.

The board with the SD shield should have an LED and series resistor connected
from pin 3 to ground. This LED will blink an error code if an error occurs.

Make sure BAUD_RATE has the same value in SerialPortLogger.pde and
SerialDataSource.pde.

Load SerialPortLogger.pde into the board with an SD shield or module and
SerialDataSource.pde into the second board.

Connect a wire between GND pins on the two boards.

Connect a wire between the serial RX pin (Pin 0) on the SD board and the
serial TX pin (pin 1) on the data source board.

It is best to power the two boards with an external 9V supply.

After the boards are powered up, wait until the pin 13 LED on the data source
board goes out. This will take up to 50 seconds.

Insert an SD into the logger board. press reset on the logger board. The
error LED connected to pin 3 should not light.

Press reset on the data source board. The pin 13 LED on the data source
board should light for about 50 seconds at 57600 baud or 25 seconds at
115200 baud while 280,000 bytes are transferred.

awesome, fat16lib!

curious how much idle time your logger has?

OP's system is not exactly twiddling its thumbs, by the sounds of things.

i am very keen to see your libraries, and have some SD card break out boards too, now. will be interesting to see what speed i can get. SPI will be good too, as I need S0 for control, S1 (115200) & 2 (9600) for serial data comms and S3 for bluetooth comms.

Updated: The '462 Image is a 328 Flattened running at 8MHz @5V. The decoupling cap is soldered to the bottom and the chip is driving a 2x16 parallel LCD... only the pot for the contrast and the reset button are required additions! I've done the same with the Nokia display which does not use the potentiometer and the reset is kinda optional... So just the 328P-PU and the bypass cap... I think I have hit rock-bottom in minimalist part usage. The software running is Magic Morse and the circuit works flawlessly from 5WPM to 40WPM.

Any chance you could post larger copies of those 2nd and 3rd images?

Certainly!

OP's system is not exactly twiddling its thumbs, by the sounds of things.

I personally would have wrote interrupt-driven UART logger what you can do is in your main loop check if a buffer reaches a certain threshold and if so

Ray says,
"It's Open Source..." It's a dedicated $2 chip... Interrupts (in user code) are really not needed to service the serial stream (at least not in my testing at 9600 BAUD.) I did not measure uC processor bandwidth, but I did throw my 9600 BAUD 'Quick Brown Fox' generator at it and did not lose any characters in the incoming stream. I built this as part of an experimental plane (Europa) engine monitoring system... the pictures are of the prototype. My friend and plane owner builds his own circuit boards so I don't waste my time anymore, but I did start using stripboard because my friend always keep my solderless breadboards (someday, they will return...)

The code may make more sense if you realize that the Mega2560 main Arduino logs output to this logger every 10 seconds (OLEDs refresh rate). The messages are diagnostic in nature but also have all of the OLED values from the engine sensors. Therefore, it was desired to always update physically to the SD with each message. Hence the reason that the buffer is checked for being full or CR:
if (c == '\r' || (bufferidx >= BUFFSIZE-1)) {

The bulk of the setup ugliness is due to managing the insertion state of the SD card and using the colored LEDs to make some sense of the states. If you do not like blinky lights, perform surgery. XD

Ray

/*
  Datalogger by M. Ray Burnette
  Designed for use in Europa Mega2560 airplane engine monitor system
  Target Atmel328P-PU @16MHz (3.3V so slightly out of spec.)
  Binary sketch size: 13,838 bytes (of a 30,720 byte maximum)
  Tested with Arduino 1.0.1
*/

#include <SD.h>

#define Red 6
#define Blue 8
#define Green 7
#define BUFFSIZE 512
#define chipSelect 10

char buffer[BUFFSIZE];
char c;
File logfile;
int bufferidx;

void(* resetFunc) (void) = 0; //declare reset function @ address 0

void setup() {

  // setup-ports for LEDS and each for 1/2 second and sync to all OFF
  pinMode( Green, OUTPUT); digitalWrite(Green, LOW); delay(500); digitalWrite(Green, HIGH);
  pinMode( Blue,  OUTPUT); digitalWrite(Blue,  LOW); delay(500); digitalWrite(Blue,  HIGH);
  pinMode( Red,   OUTPUT); digitalWrite(Red,   LOW); delay(500); digitalWrite(Red,   HIGH);

  pinMode(10, OUTPUT);  // standard convention required for SD library

  Serial.begin(9600);  // Use pins 3 Tx and 2 Rx
  delay(100);
  Serial.print(F("Serial init at 9600 BAUD\r\n"));
  Serial.print(F("(c) 2013 SofKinetics: Serial Data Logger\n\r"));

  if (!SD.begin(chipSelect)) {
    Serial.println(F("Card initialization failed!"));
    int i = millis() + 5000; // 5 seconds
    Serial.println(F("Insert card or reseat card now.  Reset pending..."));

    do {
        digitalWrite(Red, LOW); delay(250); digitalWrite(Red, HIGH); delay(250);
     } while(  i > millis() );

    Serial.println("Reset!");
    delay(250);
    resetFunc();
  }

  logfile = SD.open("europa.txt", O_CREAT | O_WRITE | O_APPEND);

  if( ! logfile ) {
    int i = millis() + 5000; // 5 seconds
    Serial.print(F("Could not open logfile \n\r"));

    do {
        digitalWrite(Red, LOW); 
        delay(250); 
        digitalWrite(Red, HIGH); 
       delay(250);
     } while( i > millis() );

    Serial.println(F("Card write failed!  Halted..."));
    do; while(true); // halt
    } 
    else {
      Serial.println(F("Writing to: europa.txt")); 
      Serial.println(F("Ready to accept serial data..."));
    }

  while (Serial.read() >= 0) {}
}

void loop() {
  digitalWrite(Blue,  HIGH); 
  digitalWrite(Green, HIGH);

  // read into program buffer, increment index

  if (Serial.available()) {
      digitalWrite(Green, LOW);
      c = Serial.read();
      buffer[bufferidx++] = c;  // Store character in array and increment index

      // CR record termination or buffer full? (CR+LF is common, so don't check both)

      if (c == '\r' || (bufferidx >= BUFFSIZE-1)) 
      {
        buffer[bufferidx] = 0; // NULL
        digitalWrite(Green, HIGH);
        digitalWrite(Blue, LOW);
        //write the program buffer to SD lib backend buffer
        logfile.write((uint8_t *) buffer, (bufferidx + 1)); 
        logfile.flush();
        bufferidx = 0;     // reset buffer pointer
      }
  }
}

The thought of using 2 x mcu in a project appeals greatly.

Actually, IMO, it makes good sense. With the Europa engine monitor, I'm using 3: One dedicated to IR receive, one dedicated to SD, and the main Mega2560.
IR with tiny85: MIM 5383H4 Infrared Module with ATtiny85 (UNO set time/date menu sample) - Exhibition / Gallery - Arduino Forum

The DS1307 is tied to the 2560, but I wanted to be able to use IR to set-up the clock and eventually the cycle-time for each sensor update; therefore I put the IR receiver off-board with a dedicated uC. All I do in the 2560 is:
if (Serial1.available() > 0) // Something from IR is in the serial buffer

There is almost no overhead to using IR in this manner with the 2560 and Serial1.

You should look at CrossRoads dual uC board, sweet: Dual ATMega328P Board! 40 IO pins. Headers to connect USB/Serial adapter, ICSP, jumpers to connect MAX232 for Serial. Cross Roads Electronics

Interesting thoughts on using multiple CPUs and I totally agree. I'm working on a system now that will/should make it easy(er) to use multiple CPUs, up to 15 boards can be addressed, each one may or may not have a CPU, it depends on the application. An SD interface is an obvious choice and it's on the drawing board.


Rob

Another obvious one is IR remote reception (and send).
http://forum.arduino.cc/index.php?topic=139907.0

I am sure many more examples exist. Anything that is a big memory buffer, requires constant polling, or requires delays() and I/O which cannot be adapted to interrupts come to mind.

Many of us sometime forget that an ATtiny running at 16MHz in RC mode has as many MIPS as a Mega2560!!! And costs less than $1.

Ray

FWIW here's a quote from my site where I rationalise the use of multiple CPUs (a "dockable" is a tiny module that performs a function for the system)

It is considered normal for a dockable to have a co-processor and they are often even used for simple functions.

Part of the rationale behind this is that processors are so cheap and powerful these days it often makes sense to use them instead of dedicated ICs.

When deciding if a dockable should use a co-processor the fundamental question to ask is "Does this dockable perform a time-consuming function that is better offloaded to another processor?"

If the answer is yes then a co-processor is probably justified. Some examples of reasonable co-processor use are...

Serial interface - A serial interface can of course be implemented with a UART, however given that a co-processor uC already has a UART, often costs half as much as a UART, and can perform many other functions such as protocol decoding it can make sense to offload this function.

Servo controller - Controlling the waveforms to 16 servos is a time-consuming job, especially if you want to apply speed control, ramp up/down etc.

IMU - Often an IMU can have an accelerometer, a compass and a magnetometer. This can be a lot of information to process, especially if floating point arithmetic is required.

Distance sensors - Ultra-sonic distance finding requires the timing of a pulse's return trip. This usually requires the processor to wait for an input to change state, better to have a co-processor wait.

Timing/counting - High-speed timing and counting usually requires the use of the processor's hardware timers but on the core processor these are often needed for other functions.

microSD - SD cards are notoriously slow to write to, especially if the data is written in small chunks. Having a co-processor performing this function allows the core processor to "write" at high speed while the co-processor handles buffering, read pre-fetching, etc.

There is an overhead to talk to a co-processor of course but no more than talking to a dedicated IO chip and sometimes even less because the interface can be at a higher, more abstract, level.


Rob

Part of the rationale behind this is that processors are so cheap and powerful these days it often makes sense to use them instead of dedicated ICs.

Yes... agree with all your points. I will also add that by using a microcontroller the software logic can be fine-tuned without having to rebuild the module. This is really important for an extra margin of flexibility and adaptability. Just ISP the unit and go through a sanity-test with the main processor.

There is an overhead to talk to a co-processor of course but no more than talking to a dedicated IO chip and sometimes even less because the interface can be at a higher, more abstract, level.

Even with serial I/O the overhead is often not a major concern, especially if the main uC is programmed to act as a state-machine. In such cases, very little time is consumed in the loop() to interrogate the buffer state and service or move on. For most programmers, this is conceptually easier and safer than trying to use interrupts to service requirements. Serial libraries are easily adjustable if necessary and double-buffering can be utilized is extremely demanding situations.

I'm working on a Christmas present: a GPS clock. The GPS module is a uC and the main uC is a 328P-PU running at 8MHz on RC. The state machine simply services the GPS as a character queue within the loop(). This is more than adequate for the default 9600 BAUD GPS. Time and Date are parsed but also there is thermistor/resistor for the temperature on the LCD. The 328 is managing the parallel LCD, doing the floating point math for the temperature, doing a Day-Of-Week calculation, and doing leap year analysis and there is uC bandwidth leftover from the $2.10 ATmel uC.

The point here is that many peripherals are already uC controlled and this is natural to us. Extending this should be a natural transition.

Ray