Fast communication to PC with low Arduino footprint

[TomS]

Hello Arduino developers!

I am working on a project where I need to sample large amounts of sensor data at high rates and send them to a computer.
I built a scaled down prototype to test everything and while it is working well so far, the speed is still an issue.

I tracked down the biggest bottleneck and found it to be the communication of the sampled data to the PC.

So far I have been using a simple Serial.print with a baudrate of 115200.
Taking into account the larger scale of the final system and the required update frequency I need something that has not only a higher data throughput but also a low very load on the Arduino.

I am looking for at least 150 KB/s and the lowest possible footprint on the Arduino performance side.

If there is a choice I'd prefer a USB based system but this is only secondary.

Can someone recommend a system that would be a good choice for these requirements?
Is there somewhere a comparison of different systems (USB-based, Ethernet, Wifi, ... ) and their footprint?


Thank you very much in advance for any help you can provide, it is much appreciated!
Have a great day!
Tom

[CrossRoads]

How about if you had SPI to a part like this
http://datasheets.maxim-ic.com/en/ds/MAX3110E-MAX3111E.pdf
and send straight serial to your PC?
Or Serial out of this chip into an FTDI chip/breakout board to have USB interface.

[robtillaart]

What is generating - 150 KB/s - on the Arduino?

Compression can do amazing things, you are talking a factor 20:1 (at least) which is quite high.
Can you provide a sample of the data e.g. 20-50 typical lines?

Alternative ethernetshield?

[TomS]

CrossRoads, I already thought that SPI to USB could possibly be a good way to go because of the low load SPI has on the Arduino. Thanks for the advice!
Is there some hardware that I could simply connect to the Arduino and then use the SPI library to talk to the computer?

robtillaart, unfortunately there is not a lot of room for compression here. The data sent mainly consists of an ID byte to identify the sensor the data is coming from and the data itself (byte or int). But thanks for the idea!

[robtillaart]

robtillaart, unfortunately there is not a lot of room for compression here. The data sent mainly consists of an ID byte to identify the sensor the data is coming from and the data itself (byte or int). But thanks for the idea!

(I do not give up easily) How many sensors are there and how much do they send?

A type of compression can be that if you send the sensor data in a fixed order the ID's can be ommitted, something like an CSV file in which you have a header line and then only lines with data. => approx 50% reduction

HEAD:  1   2   3   4   5
DATA: 10 11 10 23 24
DATA: 10 11 11 23 12
DATA: 10 12 10 23 24
DATA: 10 11 11 24 20


For 2 or more bytes values one can consider relative coding iso absolute coding (two different ways)

ABS: 1000 1001 1003 1005 1009 1004 1002 998 995 1000 (20 bytes)
REL: 1000 (2 bytes ref) 1 3 5 9 4 2 -2 -5 0  (11 bytes)  - deltas wrt reference value (value = refvalue + delta)
or
REL: 1000 (2 bytes ref) 1 2 2 4 -5 -2 -4 -3 5 (11 bytes) - deltas wrt previous value (value = prevvalue + delta)

The first one is preferred as it can handle missing bytes better than the second.


furthermore RunLengthEncoding can sometimes be interesting:

RAW: 100 100 100 100 100 100 100 100 100 100 105 106 106 106 106 106 106 (17 bytes)
RLE: 10x100 1x105 6x106 (6 bytes)

And maybe the most important question: Is all the data necessary?
I have seen apps that send data every second to the logserver only to been thrown away as the PC app analyzed per minute

Another way to get better compression is to "cook" (preprocess) sensor readings before sending them. Advantage can be high "compression" ratio's.  Drawback is that you don't have the raw data anymore.

Finally you can do some minimal noisefiltering to make the RAW data better compressable.
RAW: 100 101 102 101 102 100 101 102 101 102 102 103 104 103 104 105 104 105 105 105
FILT: 100 100 102 102 102 100 100 102 102 102 102 102 104 104 104 104 104 104 104 104
   (this filter keeps the old value as long as the new value = oldvalue +- 1. )

Rob

[graynomad]

Is there some hardware that I could simply connect to the Arduino and then use the SPI library to talk to the computer?

Yep, another Arduino.

The first one does nothing but read sensors and spit data out the SPI.

The second one does nothing but read the SPI and spit data out the USB (assuming it can do > 115200 over USB, I've not tried higher than that speed).

_____
Rob

[TomS]

Hello Rob, thank you very much for all the input!
Don't get me wrong, efficient data encoding will be very important further down the road, but I found the Serial.print to be so slow that no matter how good compression will be, I will need a faster data transmission option.

Have a look at these numbers:

Full speed sampling to internal memory (i.e. no communication): 100 Hz
Sampling + sending raw data through SPI: 73 Hz
Sampling + sending raw data through Serial.print (baudrate 115200): 2.2 Hz

As you can see Serial.print will not be an option in this application.

It will be a real time system with several hundreds of sensors connected to the Arduino. I can't give you any exact numbers as of now but we will need several Arduinos i.e. the maximum amount of sensors an Arduino will be able to sample and transmit in real time will be used.

In the end I will need an option that is able to transmit at least 150+ KB/s, possibly >1MB/s which brings me back to the question: is there some SPI to USB hardware that I could just connect to the Arduino SPI pins and that will handle the rest?

Have a nice day my friends!
Tom

[TomS]

Yep, another Arduino.

The first one does nothing but read sensors and spit data out the SPI.

The second one does nothing but read the SPI and spit data out the USB (assuming it can do > 115200 over USB, I've not tried higher than that speed).

I was thinking more along the lines of some dedicated hardware bridge, but that might be an option too, thank you!

I once tried higher baudrates and couldn't get it working although I am not sure if the Arduino or Processing was the problem.
I will try this again and see what happens.

Thanks again!
Tom

[graynomad]

Sampling + sending raw data through Serial.print (baudrate 115200): 2.2 Hz

That may have been the serial blocking, a non-blocking version might do a lot better although it will never match SPI.

______
Rob

[TomS]

That may have been the serial blocking, a non-blocking version might do a lot better although it will never match SPI.

I only read about this a few minutes ago in the context of the Teensy development board which provides up to 12Mbit/s full USB speed and non blocking serial.print calls.

So maybe a Teensy would be a good choice to read in the SPI data from the Arduino(s) and then send them to the computer.

[robtillaart]

Hello Rob, thank you very much for all the input!
Don't get me wrong, efficient data encoding will be very important further down the road, but I found the Serial.print to be so slow that no matter how good compression will be, I will need a faster data transmission option.

Hi Tom,

If you use

Code: [Select]

loop()
{
  int x = analogRead(sensorpin):
  Serial.println(x);
}
you are translating all integers to strings to text, so the number 1001 will be "1001\r\n" That is six bytes,

Check the Serial.write(x,2);  it will send the int as two bytes  six/two = three times faster or better said  6/2=3x faster

115200 can send max 50000 ints per second, lets say 40K in practice that is not much but I would like to know some background about what is generated.

The Arduino can call analogRead(A0) less than 10.000 times per second (20Kbytes)  and  digitalRead()  200.000 times per second (25Kbytes) 
so unless you do lowlevel reading of 8 IO lines in parallel e.g. registerB the serial port should be able to handle it.

Can you share a sample of the  (uncompressable)  data or the tell us more about the data that is generated?

[robtillaart]

Time todo a simple serial port test

a simple sketch Arduino 2009 IDE 22; <--> windows 7 64 bit with  putty.exe

Code: [Select]


void setup()
{
 Serial.begin(345600);   // !! 115200, 230400, 345600,   460800 X
}

void loop()
{
 unsigned long t1 = millis();
 for (long i=0; i< 10000; i++)
 {
  Serial.println("12345678");  // 10 bytes incl \r\n  
 }
 Serial.println(millis() - t1);
}

some runs.
100K chars took 8501 msec @ 115200 baud = 11763 bytes/second (IDE sermon)
100K chars took 4500 msec @ 230400 baud = 22222 bytes/second (used putty.exe)
100K chars took 2999 msec @ 345600 baud = 33333 bytes/second (used putty.exe)

460800 baud failed ...

Serial.write() was not faster in # bytes.

So there might be hope ...

-- update --
Non standard baudrates
100K chars took 2499 msec @ 400000 baud = 40016 bytes/second (used putty.exe)
100K chars took 1999 msec @ 500000 baud = 50025 bytes/second (used putty.exe)

600000 baud failed

no guarantees but there seems to be room to play ...

500Kbaud is (sometimes) better than 9600 .... e.g. Arduino's 2K RAM can be send in approx 41 millisec.

[CrossRoads]

TomS,
I don't know if any add-on boards exist.
Making a card with the Maxim chip and the FTDI chip would be pretty straightforward.

You could also browse here
http://shieldlist.org/
and see if something exists already.

[TomS]

Hello Rob, thanks a lot for showing us these test results!
Maybe you could make them public for all on the Serial reference page, I am sure others are interested in these numbers as well!

For now I ordered a Teensy 2.0 since I will need a central place from which to share the data of several Arduinos to the computer anyway, and I will get back to this problem and see how the Teensy Serial performs once it arrives.

Thanks again CrossRoads, Graynomad and Rob for all the advice, it is much appreciated!

Have a great day!
Tom

[westfw]

a low very load on the Arduino.

All of the atmega devices (i2C, SPI, Serial) will end up having essentially interrupt (or block) per byte to send data, regardless of speed.  At high speed, none of them are going to be "low load", and they should be approximately similar to each other wrt "load" on the arduino.

I can't understand why your function slows down to 2.2Hz using serial.  1/2.2s should be enough time to transmit over 5000 bytes of data at 115200, though of course transmitting more that 150kbps on a 115kbps link is going to be impossible.