Go Down

Topic: Fast communication to PC with low Arduino footprint (Read 7 times) previous topic - next topic

TomS

Hello Arduino developers!

I am working on a project where I need to sample large amounts of sensor data at high rates and send them to a computer.
I built a scaled down prototype to test everything and while it is working well so far, the speed is still an issue.

I tracked down the biggest bottleneck and found it to be the communication of the sampled data to the PC.

So far I have been using a simple Serial.print with a baudrate of 115200.
Taking into account the larger scale of the final system and the required update frequency I need something that has not only a higher data throughput but also a low very load on the Arduino.

I am looking for at least 150 KB/s and the lowest possible footprint on the Arduino performance side.

If there is a choice I'd prefer a USB based system but this is only secondary.

Can someone recommend a system that would be a good choice for these requirements?
Is there somewhere a comparison of different systems (USB-based, Ethernet, Wifi, ... ) and their footprint?


Thank you very much in advance for any help you can provide, it is much appreciated!
Have a great day!
Tom

CrossRoads

How about if you had SPI to a part like this
http://datasheets.maxim-ic.com/en/ds/MAX3110E-MAX3111E.pdf
and send straight serial to your PC?
Or Serial out of this chip into an FTDI chip/breakout board to have USB interface.
Designing & building electrical circuits for over 25 years. Check out the ATMega1284P based Bobuino and other '328P & '1284P creations & offerings at  www.crossroadsfencing.com/BobuinoRev17.
Arduino for Teens available at Amazon.com.

robtillaart


What is generating - 150 KB/s - on the Arduino?

Compression can do amazing things, you are talking a factor 20:1 (at least) which is quite high.
Can you provide a sample of the data e.g. 20-50 typical lines?

Alternative ethernetshield?


Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

TomS

CrossRoads, I already thought that SPI to USB could possibly be a good way to go because of the low load SPI has on the Arduino. Thanks for the advice!
Is there some hardware that I could simply connect to the Arduino and then use the SPI library to talk to the computer?

robtillaart, unfortunately there is not a lot of room for compression here. The data sent mainly consists of an ID byte to identify the sensor the data is coming from and the data itself (byte or int). But thanks for the idea!

robtillaart

Quote
robtillaart, unfortunately there is not a lot of room for compression here. The data sent mainly consists of an ID byte to identify the sensor the data is coming from and the data itself (byte or int). But thanks for the idea!


(I do not give up easily) How many sensors are there and how much do they send?

A type of compression can be that if you send the sensor data in a fixed order the ID's can be ommitted, something like an CSV file in which you have a header line and then only lines with data. => approx 50% reduction

HEAD:  1   2   3   4   5
DATA: 10 11 10 23 24
DATA: 10 11 11 23 12
DATA: 10 12 10 23 24
DATA: 10 11 11 24 20


For 2 or more bytes values one can consider relative coding iso absolute coding (two different ways)

ABS: 1000 1001 1003 1005 1009 1004 1002 998 995 1000 (20 bytes)
REL: 1000 (2 bytes ref) 1 3 5 9 4 2 -2 -5 0  (11 bytes)  - deltas wrt reference value (value = refvalue + delta)
or
REL: 1000 (2 bytes ref) 1 2 2 4 -5 -2 -4 -3 5 (11 bytes) - deltas wrt previous value (value = prevvalue + delta)

The first one is preferred as it can handle missing bytes better than the second.


furthermore RunLengthEncoding can sometimes be interesting:

RAW: 100 100 100 100 100 100 100 100 100 100 105 106 106 106 106 106 106 (17 bytes)
RLE: 10x100 1x105 6x106 (6 bytes)

And maybe the most important question: Is all the data necessary?
I have seen apps that send data every second to the logserver only to been thrown away as the PC app analyzed per minute

Another way to get better compression is to "cook" (preprocess) sensor readings before sending them. Advantage can be high "compression" ratio's.  Drawback is that you don't have the raw data anymore.

Finally you can do some minimal noisefiltering to make the RAW data better compressable.
RAW: 100 101 102 101 102 100 101 102 101 102 102 103 104 103 104 105 104 105 105 105
FILT: 100 100 102 102 102 100 100 102 102 102 102 102 104 104 104 104 104 104 104 104
   (this filter keeps the old value as long as the new value = oldvalue +- 1. )

Rob
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Go Up