FYI: Serial Data Logger

Just got a PCB from Jack Christensen’s up and running and thought others would be interested.
http://forum.arduino.cc/index.php?topic=252905.0
Basically, you connect the TX (D1) pin on your Arduino project to the “Serial Data Logger” RX input pin.
Your Arduino project would be programmed to send output using Serial.print().
Whatever is sent out is received by the “Serial Data Logger” and then written to the data logger’s SD card.
This method allows your Arduino project to do data logging without having to have SD sketch code or hardware on it.
Some possibilities are:

  • Send sensor data to a “Serial Data Logger” CSV file. Use Excel for analysis, graphing, etc.
  • Send de-bug data to a “Serial Data Logger” SD file for logging.
    With a data logger like this, you do not need to always add hardware and software to a project, saving money, programming memory, and complexity.
    Some features of the “JC Serial Data Logger” PCB are:
  • Download new software either with ICSP or FTDI headers.
  • Switch adjustable baud rate.
  • Control switch for starting and stopping logging.
  • Spring-loaded Mico SD card socket (I am using a Kingston 4GB card).
  • Output indicators for: Overrun, SD Activity, and Heart Beat LEDS.
  • Power jack.
  • Power input select jumper.
  • Small size.
    Note: Jack has released his software for download so you can build your own logger out of a standalone chip or an Arduino UNO, Bobuino2 etc.
    I know I will be using this often.
    Thanks Jack

Edit, added PCB imge
serial_logger_pcb.jpg

Larry, thanks for the props. Mine are working well, I hope yours do too! Mine seem to be able to log a continuous data stream at 57600 baud without data loss. They may not keep up with 115200, but I should try that again. The two that I have running are indeed operating at 115200, but the data stream has pauses in it, so no problems with that. Of course YMMV based on the characteristics of the particular SD card used.

In the diagram, why isn't the gnd wire [the "red" one!] going to the gnd pin on the FTDI header?

Jack, how are you buffering the incoming data, and deciding when to write it to the SD card? I ask because apparently SD cards always read and write 512-bytes at a time [I believe].

The two pins on the right hand side of the FTDI header are both GND.

When the buffer reaches 512 it is written to the SD card.
I believe Jack wrote the code so it is doubled buffered.
Also when you push the Start/Stop switch the remainig data is sent to the SD card and the file is closed.

Edit: added an image of the data logger put in an enclosure.
I am using a USB power supply to power the unit.

2014-11-23_20-53-42.jpg

oric_dan: In the diagram, why isn't the gnd wire [the "red" one!] going to the gnd pin on the FTDI header?

My bad, just grabbed some wires. It's actually orange but may have been influenced in the imaging process. I know that the negative electricity likes black wires better and I am usually more careful about that.

Jack, how are you buffering the incoming data, and deciding when to write it to the SD card? I ask because apparently SD cards always read and write 512-bytes at a time [I believe].

Using the USART_RX interrupt. There are two input buffers, each 512 bytes. The ISR puts the incoming character into the next available position in the current buffer. Once the buffer is full, a flag is set and the buffers are switched. The mainline code (i.e. non-ISR), watches for the buffer full flags and writes full buffers to the SD card. After being written, the buffer is marked empty. If after switching buffers, the next buffer is found not to be empty, then that is an overrun situation. And as Larry said, pushing the Start/Stop button flushes the buffers to the SD card.

I started testing with smaller buffers, but was happy to find that I had enough SRAM for 2x512 byte buffers. There's not a lot of SRAM left, something less than 200 bytes, but it's evidently enough.

Hmm, the color seen depends on the monitor. Looks red on my tablet, but orange on my PC monitor. [BTW, in line with that other thread, the black wire is always the path of least resistance].

Double-buffering, cool. Offhand do you know how long it takes to write the 512-buffer to the SD card? I was wondering about your streaming rate. At 57600, it should take about (10/57600)*512 = 89 msec to send over 512 bytes. I'd have thought the SPI transfer plus write time to SD would be a lot faster than that.

BTW, looking at your schematic, there is a potential problem since you've wired the Vcc pin on the FTDI header. I keep that pin free on all of my own boards. The problem is, if you plug in the FTDI adaptor and forget to pull the Vin plug, you can potentially blow the USB port on the computer that's supplying the FTDI Vcc.

I actually did that on a jeenode once upon a time, but there, they have the FTDI Vcc wired directly to Vin, which is many levels of debacle. http://mark.mckillen.com/wp-content/uploads/2013/03/KiCAD-versoin-of-JeeNodeV6-Schematic.png http://jeelabs.net/attachments/download/330/jlpcb-128.png

oric_dan: Offhand do you know how long it takes to write the 512-buffer to the SD card?

Unknown. But it probably depends on the card to some extent.

At 57600, it should take about (10/57600)*512 = 89 msec to send over 512 bytes.

Sounds right.

I'd have thought the SPI transfer plus write time to SD would be a lot faster than that.

I would think usually so, but from reading some of what @fat16lib has written, my understanding is that due to internal goings-on in the SD card, the occasional I/O can take much longer than average.

BTW, looking at your schematic, there is a potential problem since you've wired the Vcc pin on the FTDI header. I keep that pin free on all of my own boards. The problem is, if you plug in the FTDI adaptor and forget to pull the Vin plug, you can potentially blow the USB port on the computer that's supplying the FTDI Vcc.

I frequently want to power my projects from the USB port via the FTDI adapter. In the case of this logger, it will often be powered by the project it's monitoring. Hooking up two power supplies simultaneously is of course a big no-no but I suppose not everyone knows that or occasionally forgets. I imagine that I have forgotten too, but have not had a problem. USB ports usually have some overcurrent protection, but I'm also in the habit of using a separate powered hub. I also have a couple FTDI adapters, one with the strap cut so that it doesn't supply power.

At any rate it is a point well taken, TYVM. An SPDT switch or just a three-pin header with a jumper to select power from one source or the other would be potential solutions. I'll make a note for v2.1.

A 3-pin header with a movable shunt sounds about foolproof. Would have saved my USB port. Actually, I didn't totally fry the port, it just tripped some internal ckt breaker in my notebook, and shut the computer off completely. Then, it wouldn't restart for a while.

I only started playing with the SD library a couple of weeks ago for the first time, so am mostly postulating here:

  1. if you're using the IDE library, remember it's 5 years old, and Bill Greiman's newer SdFat libraries may be more efficient.

  2. actually I think you may be doing "quadruple" buffering here.

The SD library uses its own 512-byte buffer, and as I understand it, it works by reading in an entire 512-byte sector from the SD card, patching in the new data to be written, and then writing the 512-byte buffer back to the card.

So, it seems to me that, if you're writing a lot of data, then the library will do multiple buffer fills and re-writes, as you write off the end of one sector and into the start of the next one. IOW, if you fill up your own 512-byte streaming buffer before writing to the SD card, my guess is you'll be writing past the end of a card sector almost 100% of the time. Stated another way, your data will likely not write into the card at exact sector boundaries, and the library will be doing 2 buffer fills and rewrites for every one of your streaming buffer writes.

So, it actually may be more efficient for you to make your streaming buffer "smaller", say 384-bytes. Then it will do quadruple-buffering only about every other time, rather than every time. Just a hypothesis, but would be easy to test.