I am working on a project where I need to sample large amounts of sensor data at high rates and send them to a computer.
I built a scaled down prototype to test everything and while it is working well so far, the speed is still an issue.
I tracked down the biggest bottleneck and found it to be the communication of the sampled data to the PC.
So far I have been using a simple Serial.print with a baudrate of 115200.
Taking into account the larger scale of the final system and the required update frequency I need something that has not only a higher data throughput but also a low very load on the Arduino.
I am looking for at least 150 KB/s and the lowest possible footprint on the Arduino performance side.
If there is a choice I'd prefer a USB based system but this is only secondary.
Can someone recommend a system that would be a good choice for these requirements?
Is there somewhere a comparison of different systems (USB-based, Ethernet, Wifi, ... ) and their footprint?
Thank you very much in advance for any help you can provide, it is much appreciated!
Have a great day!
Tom
Compression can do amazing things, you are talking a factor 20:1 (at least) which is quite high.
Can you provide a sample of the data e.g. 20-50 typical lines?
CrossRoads, I already thought that SPI to USB could possibly be a good way to go because of the low load SPI has on the Arduino. Thanks for the advice!
Is there some hardware that I could simply connect to the Arduino and then use the SPI library to talk to the computer?
robtillaart, unfortunately there is not a lot of room for compression here. The data sent mainly consists of an ID byte to identify the sensor the data is coming from and the data itself (byte or int). But thanks for the idea!
robtillaart, unfortunately there is not a lot of room for compression here. The data sent mainly consists of an ID byte to identify the sensor the data is coming from and the data itself (byte or int). But thanks for the idea!
(I do not give up easily) How many sensors are there and how much do they send?
A type of compression can be that if you send the sensor data in a fixed order the ID's can be ommitted, something like an CSV file in which you have a header line and then only lines with data. => approx 50% reduction
And maybe the most important question: Is all the data necessary?
I have seen apps that send data every second to the logserver only to been thrown away as the PC app analyzed per minute
Another way to get better compression is to "cook" (preprocess) sensor readings before sending them. Advantage can be high "compression" ratio's. Drawback is that you don't have the raw data anymore.
Finally you can do some minimal noisefiltering to make the RAW data better compressable.
RAW: 100 101 102 101 102 100 101 102 101 102 102 103 104 103 104 105 104 105 105 105
FILT: 100 100 102 102 102 100 100 102 102 102 102 102 104 104 104 104 104 104 104 104
(this filter keeps the old value as long as the new value = oldvalue +- 1. )
Hello Rob, thank you very much for all the input!
Don't get me wrong, efficient data encoding will be very important further down the road, but I found the Serial.print to be so slow that no matter how good compression will be, I will need a faster data transmission option.
Have a look at these numbers:
Full speed sampling to internal memory (i.e. no communication): 100 Hz
Sampling + sending raw data through SPI: 73 Hz
Sampling + sending raw data through Serial.print (baudrate 115200): 2.2 Hz
As you can see Serial.print will not be an option in this application.
It will be a real time system with several hundreds of sensors connected to the Arduino. I can't give you any exact numbers as of now but we will need several Arduinos i.e. the maximum amount of sensors an Arduino will be able to sample and transmit in real time will be used.
In the end I will need an option that is able to transmit at least 150+ KB/s, possibly >1MB/s which brings me back to the question: is there some SPI to USB hardware that I could just connect to the Arduino SPI pins and that will handle the rest?
The first one does nothing but read sensors and spit data out the SPI.
The second one does nothing but read the SPI and spit data out the USB (assuming it can do > 115200 over USB, I've not tried higher than that speed).
I was thinking more along the lines of some dedicated hardware bridge, but that might be an option too, thank you!
I once tried higher baudrates and couldn't get it working although I am not sure if the Arduino or Processing was the problem.
I will try this again and see what happens.
Graynomad:
That may have been the serial blocking, a non-blocking version might do a lot better although it will never match SPI.
I only read about this a few minutes ago in the context of the Teensy development board which provides up to 12Mbit/s full USB speed and non blocking serial.print calls.
So maybe a Teensy would be a good choice to read in the SPI data from the Arduino(s) and then send them to the computer.
Hello Rob, thank you very much for all the input!
Don't get me wrong, efficient data encoding will be very important further down the road, but I found the Serial.print to be so slow that no matter how good compression will be, I will need a faster data transmission option.
Hi Tom,
If you use
loop()
{
int x = analogRead(sensorpin):
Serial.println(x);
}
you are translating all integers to strings to text, so the number 1001 will be "1001\r\n" That is six bytes,
Check the Serial.write(x,2); it will send the int as two bytes six/two = three times faster or better said 6/2=3x faster
115200 can send max 50000 ints per second, lets say 40K in practice that is not much but I would like to know some background about what is generated.
The Arduino can call analogRead(A0) less than 10.000 times per second (20Kbytes) and digitalRead() 200.000 times per second (25Kbytes)
so unless you do lowlevel reading of 8 IO lines in parallel e.g. registerB the serial port should be able to handle it.
Can you share a sample of the (uncompressable) data or the tell us more about the data that is generated?
Hello Rob, thanks a lot for showing us these test results!
Maybe you could make them public for all on the Serial reference page, I am sure others are interested in these numbers as well!
For now I ordered a Teensy 2.0 since I will need a central place from which to share the data of several Arduinos to the computer anyway, and I will get back to this problem and see how the Teensy Serial performs once it arrives.
Thanks again CrossRoads, Graynomad and Rob for all the advice, it is much appreciated!
All of the atmega devices (i2C, SPI, Serial) will end up having essentially interrupt (or block) per byte to send data, regardless of speed. At high speed, none of them are going to be "low load", and they should be approximately similar to each other wrt "load" on the arduino.
I can't understand why your function slows down to 2.2Hz using serial. 1/2.2s should be enough time to transmit over 5000 bytes of data at 115200, though of course transmitting more that 150kbps on a 115kbps link is going to be impossible.
Dear folks,
is there any way to override connection baud rate via usb programming port in windows or Mac?
i set the baud rate in Arduino Due to Serial.begin(128000) and also in PC to 128000 and everything is ok!
But for higher baud rates i cant get sensible output via hyper terminal or putty.
something like the attachment below
Best results are if the baud rate is a divider of the clock frequency of the chip.
For an UNO this is 16MHZ so 500K and 250K will work better than 256K
No experience with Due but I expect similar "preferences"
Putty should have no problems with 250K and 500K under windows 7.
There are other free terminals like realterm that also do a good job.