# Speed of Data logging to PC, buffering, timing and jitter

Hi,
Absolute beginner at hardware programming here.
After several days of searching for answers online I still have a couple of basic questions :

a) I understand that when logging acquired data (analog or digital read) to a PC from an Arduino, two factors (at least) come into play - 1) sampling speed 2) transfer speed. The sampling is quite sufficient for my application (I need to sample something at around 20 kHz).
The transfer speeds (stemming from inherent Serial sluggishness?) however reduce the total rate to about 1.7 kHz.
I have tried buffering, and transferring the buffer (e.g. of about 1000 samples). This results in extremely frequent samples (I have seen speeds around 200 kHz on my Due) within the buffer itself but rather hefty gaps between the buffers. I guess this makes sense since the larger the buffer, the more time to transfer it. Such a scheme is suboptimal, since the data with such gaps is not very useful. So here is the thing which I'm trying to understand on a fundamental level - I've heard of double-buffering, where one buffer is sampled, and another transferred. Unless I misunderstood the context of double-buffering, it only makes sense if the two actions can be performed simultaneously - so here is the question - can Arduino (or other ADC boards) sample and transfer at the same time, or is the CPU busy transferring and cannot therefore sample until the transfer is done? If so, what is the advantage of double buffering.

b) Transferring the values measured by the sensor over serial takes long enough, but I'm transferring also the time stamp of each acquisition (I would ideally like them to take place at equal intervals). I would love not to have to transfer the time stamps as well, which would reduce the bits transferred and speed up the process. Instead I could send a time-stamp say once in every 10 000 samples, then divide the interval by 10 000 to calculate individual time-stamp values. However I am not sure whether I can trust that the intervals are regular. If the jitter is inherent, how do I make sure the sample intervals are more regular. Artificially introduce delays?

Thank you in advance and please forgive the verboseness of my post.

The bottleneck is ALWAYS going to be how long it takes to send data via the serial port. It does not matter whether you collect the samples in two arrays, one array, or 20000 scalar variables. Nothing will make serial go any faster.

The only way to ensure that you get consistent intervals is to forget about trying to get consistent intervals.

Read a value. Send it to the serial port (actually to the outgoing buffer). Then, wait until the data is gone. Repeat for the next sample.

The receiving end can time stamp the data as it arrives. It won't necessarily arrive at constant intervals, but there is a reasonable assumption that the latency between sending the value and it being received will be relatively constant.

How are you sending the data? As text or as binary data?

Text, I think.. Basically just following the simplest examples out there.
So from what I gather my only choice of getting a decent data-logging rate out of this is getting a Teensy?
The consensus seems to be that it provides a much higher transfer rate.

The consensus seems to be that it provides a much higher transfer rate.

It can't send serial data any faster than the Arduino can.

PaulS:
It can't send serial data any faster than the Arduino can.

I'm not sure what MCU is in the Teensy. However a Leonardo (which does not use the USART for communication with the PC) can work at full USB speed with Arduino 1.5.7. The baud rate is ignored.

The USB system (regardless of whether a Leonardo or an Uno) is very poor for shifting data a few bytes at a time. The best throughput will be with larger chunks - perhaps 64 bytes and up.

...R

I'm curious about what you are sending that requires such a rapid throughput. Could you perhaps give some more details of your project. Maybe we can come up with an alternative method that reduces the demands on the serial bus.

I am basically building a tachometer. I have two LED-Photodiode pairs to measure rate and direction of rotation of a turbine (the flaps of the turbine block/unblock the LED-photodiode, that create a HIGH or LOW state on the digital input pin). The algorithm itself is quite simple and works well at low rpms, but apparently during certain periods the turbine rotates fast enough that the data stops looking meaningful.
I tried it on Leonardo and Due, both of which supposedly support "true" USB, but best I've seen (Due) is 1.7kHz.

So what DATA is actually being sent? So, based on the output from these sensors, the arduino can determine the rate and direction of the turbine BUT why does it have to send the info so often?

Well, the idea is to have real-time monitoring of the rotation rate. If that rate is on the order of several kHz, then by Nyquist I have to sample it at least at twice that rate (I'm aiming for around 20kHz). The computer (ideally Matlab) is making the actual rate calculation etc. In other words the arduino is sending 1's and 0's according to whether the digital pins see enough voltage or not. It is also sending the corresponding timestamps. Based on this data the rate is calculated by the computer.
I am considering performing the calculation on the Arduino itself and only sending the result out, but it would still need to be pretty frequent (although not nearly as frequent as sending the raw data). The only reservation I have is that potentially performing the calculation on the Arduino will slow down the process as well. Not sure which option would be best.

Nyquist? But you have photo diodes! These produce a digital pulse NOT a sinusoidal wave. Nyquist is totally irrelevant.

mischki:
the flaps of the turbine

Sounds like a real hi-tech project

I can't see why you need to send individual pulses over the USB connection. Just get the Arduino to measure the time for 100 or 1000 pulses (or more) and pass the result to the PC.

...R

KenF:
Nyquist? But you have photo diodes! These produce a digital pulse NOT a sinusoidal wave. Nyquist is totally irrelevant.

Wouldn't you still need to check if a pin is HIGH or LOW faster than it would be expected to change states? Otherwise, you risk not seeing a state change right?