ArdunoMega2540 -> Ethernet Shield... UDP latency

Projects Networking, Protocols, and Devices
1

Hello,

I need your help. I've been fighting a latency issue and can't find the cause.

I have serial device (encoder motor) sending data at 115200 baud (the actual data consists of only 888 bytes per second)

I need to deliver this data over local network to two computers (raspberry and mac) with minimum latency.
However data arrives at both with 40ms lag.

My setup is following

SerialDevice [115200baud] -> Ardiuno -> Shield ---[UDP] ---> Gigabit Router -> Raspberry (and Mac)

Reading serial data seems to be not a problem. To prove that it's not, I blink a led when I reach certain encoder value (hard coded in my sketch). It blinks exactly when motor reaches this position, I checked by filming the setup with a video camera.

However, when I run on raspberry "nc -uklvw 0 port" to recover UDP packets, I observe this 40ms lag from the moment rapsberry "blinks" and value appearing in my terminal.

Actually it looks like this issue goes beyond Arduino. For testing purposes I replaced Arduino/Shield combo with a a specialized hardware converter RS232->Ethernet friends have borrowed me. It called SDS4, costs 400 euros and made by Perle.

To my surprise I have the exactly same lag.

40ms seems a lot ( I have visual application and this lag is obvious to anyone). I've seen some articles about high-speed UDP applications and they talk about 4-5 ms latency.

Any ideas ?

Thanks
M

2

Have you tried using Wireshark or similar to see the packets on the ethernet and the timings?

Is the data being sent as a single packet, or does it end up being sent at a byte per packet? With Ethernet the minimum packet size is 64bytes, so if there is insufficient data to fill the packet it is padded out to the minimum size. Arduino ethernet runs at 10Mb/s so just over 29000 packets per second at the minimum size, which would take just over 33ms for 888 packets assuming no other delays.

3

It appears you are trying to use real time networking which is a difficult task. The first thing you need to do is to analyze your, system connectivity requirements. The right way to do it is to map and classify traffic in the system. First classification to be made is messages sizes and their latency requirements. In many cases, these two parameters are sufficient to choose transport protocols and network topology. Try this link for more informaiton: Real time communications over UDP protocol - CodeProject

4

countrypaul:
Have you tried using Wireshark or similar to see the packets on the ethernet and the timings?

Is the data being sent as a single packet, or does it end up being sent at a byte per packet? With Ethernet the minimum packet size is 64bytes, so if there is insufficient data to fill the packet it is padded out to the minimum size. Arduino ethernet runs at 10Mb/s so just over 29000 packets per second at the minimum size, which would take just over 33ms for 888 packets assuming no other delays.

very interesting information.
Data is being send in a single packet as soon as I receive it, I send it out.
It's 5 bytes. Ok, so if it's padded it's 64, but shouldn't be a problem.
I don't have 888 packets, but 888 bytes per second.
Basically I receive 150 encoder readings per second and each is 6 characters long (plus occasional control characters).
So the data load is very light, however the question here is how fast I can transport these 6 bytes across the network.

5

gilshultz:
It appears you are trying to use real time networking which is a difficult task. The first thing you need to do is to analyze your, system connectivity requirements. The right way to do it is to map and classify traffic in the system. First classification to be made is messages sizes and their latency requirements. In many cases, these two parameters are sufficient to choose transport protocols and network topology. Try this link for more informaiton: Real time communications over UDP protocol - CodeProject

Thanks for the link. Quite interesting reading. I can see that average RTT for UDP-RT is 10 ms, and maximum is 50. It's better that what I have for sure.
However in terms of network mine is very simple, so I don't see how I can optimize it further
Arduino, Raspberry and Mac are all plugged in a gigabit switch.
There is no other traffic between them. And as I mentioned above the dataload is also extremely light.
Merely 7 bytes 150 times a second.

Still puzzling.

Yes, real-time is tricky.

6

Is the data one directional0? in that does the Arduino just send a packet to the RPi or does it expect a reply?

The Arduino ethernet interface is only 10Mb/s so that is likely to be the limiting network factor and in practice it is difficult to get more than about 3Mb/s throughput.

Which RPi are you using? The 3 has 100Mb/s interface and the 4 1Gb/s.

How long are the cables joining the Arduino to the switch, and also the RPi to the switch?

Have you tried missing out the switch and having a direct cable Arduino to RPi?

You say "However, when I run on raspberry "nc -uklvw 0 port" to recover UDP packets, I observe this 40ms lag from the moment rapsberry "blinks" and value appearing in my terminal." What software are you using on the RPi to view the daa on the terminal?

7

countrypaul:
Is the data one directional0? in that does the Arduino just send a packet to the RPi or does it expect a reply?

unidirectional. arduino just notifies RPi (and Mac) about current encoder position.

countrypaul:
The Arduino ethernet interface is only 10Mb/s so that is likely to be the limiting network factor and in practice it is difficult to get more than about 3Mb/s throughput.

really ? I thought it's 10/100mbit. when I check the router I see 100mbit connection from Arduino Shiled.

countrypaul:
How long are the cables joining the Arduino to the switch, and also the RPi to the switch?

very short. They are sitting next to each other.

countrypaul:
Have you tried missing out the switch and having a direct cable Arduino to RPi?

no I didn't. but I tried this with OSX and it didn't make any difference.

countrypaul:
You say "However, when I run on raspberry "nc -uklvw 0 port" to recover UDP packets, I observe this 40ms lag from the

moment rapsberry "blinks" and value appearing in my terminal." What software are you using on the RPi to view the daa on the terminal?

hmm.. I use just cmd terminal.. and nc...
I then take a video of arduino next to my screen and I see the latency.
Thanks
M

8

Sorry, I confused two threads, the ENC28J60[color=#202124][tt] [/color][/tt]is 10Mb/s the ws5100, 5200, 5500 are 10/100Mb/s. Which do you have?
In fact 3Mb/s is optimistic, it appears 300Kb/s is often quoted as the maximum many get put through onto Ethernet.

Either way that should be sufficient for your use, even given you are sending the data to 2 destinations rather than one.

The latency you describe appears to be entirely on the RPi, given the led blinks on the RPi and then the data appears on the cmd terminal. I had assumed the led was on the Arduino.

If you ping the Arduino from the RPi what figures are returned?

It may be that you need to find another piece of software on the RPi or possibly write your own, assuming I ave understood everything correctly (50/50 chance perhaps!)

9

countrypaul:
Sorry, I confused two threads, the ENC28J60 is 10Mb/s the ws5100, 5200, 5500 are 10/100Mb/s. Which do you have?
In fact 3Mb/s is optimistic, it appears 300Kb/s is often quoted as the maximum many get put through onto Ethernet.

Either way that should be sufficient for your use, even given you are sending the data to 2 destinations rather than one.

The latency you describe appears to be entirely on the RPi, given the led blinks on the RPi and then the data appears on the cmd terminal. I had assumed the led was on the Arduino.

If you ping the Arduino from the RPi what figures are returned?

It may be that you need to find another piece of software on the RPi or possibly write your own, assuming I ave understood everything correctly (50/50 chance perhaps!)

I have the 100mbit one.
You are correct, the LED is on arduino.
Let me explain again.

  • I have arduino, which converts motor encoder serial line to UDP (broadcast)
  • I have to consumers: RPi and Mac both plugged into the same switch.
    I have observed a latency at which data arrives to both RPi and Mac. I thought maybe there is a problem with serial line. So I setup a physical test and made Arduino to blink when it reads from serial port a specific encoder position. It does blink at the exact right moment. That proves that serial data arrives with no buffering/delays/lag.
    However the UDP data arrives way too late. And I don't really understand why. In the spirit of previous test, I decided not to send udp data continously, but only when this specified encoder position is reached just once. Sure enough - it arrives 33ms late.
    Probably I can simplify test. Just make arduino blink and send some data via udp. I'm sure data will arrive 30ms later.
    I observe this on RPi and Mac. I've changed router. I've switched cables.

M

ps. here is the link to the video, to give the idea of what this lag look like

10

Can you just confirm where is the LED that blinks? In your first post you said the Raspberry blinks, but in your latest post you say the LED is on the Arduino.
There are several different Ethernet shields that operate at 10/100 Mb/s do you know which you have?

I know the Nagle algorithm is used with TCP to collect a number of small amounts of data into 1 packet rather than sending several small ones. If this also applies to UDP which could be a reason for the delay over ethernet - it being caused by waiting for more data before actually sending what it has - however I didn't think this applied to UDP.

One way to test this might be to try sending say slightly under 1500 bytes of data (Ethernet MTU is 1500 and UDP overhead is about 24bytes) and see if there is still a delay in sending it.

I can't see anything on the movie file you have posted - are you sure that is the right link?

11

countrypaul:
Can you just confirm where is the LED that blinks? In your first post you said the Raspberry blinks, but in your latest post you say the LED is on the Arduino.
There are several different Ethernet shields that operate at 10/100 Mb/s do you know which you have?

I know the Nagle algorithm is used with TCP to collect a number of small amounts of data into 1 packet rather than sending several small ones. If this also applies to UDP which could be a reason for the delay over ethernet - it being caused by waiting for more data before actually sending what it has - however I didn't think this applied to UDP.

One way to test this might be to try sending say slightly under 1500 bytes of data (Ethernet MTU is 1500 and UDP overhead is about 24bytes) and see if there is still a delay in sending it.

I can't see anything on the movie file you have posted - are you sure that is the right link?

Correct. It's Arduino (if i told previously it's RPi, it's my mistake), which blinks.
The Shield I have is W5100.
Here is my loop btw:
void loop() {

length1 = Serial1.readBytesUntil('-', buffer1,100);
buffer1[length1+1]='\0';
if (!reached && buffer1[1] == '4' && buffer1[2] == '8' && buffer1[3] == '1' && buffer1[3] == '0') {
digitalWrite(LED_BUILTIN, HIGH); // turn the LED on
reached = true;
Udp.beginPacket(broadcast, localPort);
Udp.write(buffer1, length1);
Udp.endPacket();
}
(serial feed is ascii it looks like "...04775-04777-014779-04781...")
here I check the characters from serial buffer datagram if it's value is 4810.
If it is => blink and send out UDP packet.
I recover it on Rpi and Mac way too late.
As for UDP waiting for packet to be complete... isn' what Udp.endPacket(); is doing ?
Or you are talking about lower-level UDP implementation ?
But doing this test might be interesting, I agree. And not too complicated.
Movie I've posted is the final form of what I'm doing. My motors are winches and they pull panels. I use videoprojector to project image in real-time, based on encoders. The image is trailing behind due to this UDP delay. At some moment it's paused and you see it more clearly.

Thanks
M

12

This is not directly related to the problem, but I am a little perplexed by the line
if (!reached && buffer1[1] == '4' && buffer1[2] == '8' && buffer1[3] == '1' && buffer1[3] == '0') {

I assume that is equivalent to

if (!(reached && (buffer1[1] == '4') && (buffer1[2] == '8') && (buffer1[3] == '1') && (buffer1[3] == '0'))) {

It was the presence of buffer1[3] that caught my eye.

I tend to always put brackets in place as I've seen too many people fall foul of the precedence or operators.

In your code what is the definition and value of broadcast?
13

One thing I have not asked is whether the delay applies to all packets or just the first? If it is just the first then it could be down to MAC address resolution.

I think installing wireshark on either the RPi or the MAC might help to determine whether the packets appear on the RPi (MAC) soon after arriving or are delayed.

Nothing we have noted appears to clarify where the delay is. All we seem to have clarified is that whether you use an Arduino with Ethernet shield or a hardware RS232 to Ethernet converter the RPi (MAC) appears to get it 40s late.

14

countrypaul:
This is not directly related to the problem, but I am a little perplexed by the line
if (!reached && buffer1[1] == '4' && buffer1[2] == '8' && buffer1[3] == '1' && buffer1[3] == '0') {

I assume that is equivalent to

if (!(reached && (buffer1[1] == '4') && (buffer1[2] == '8') && (buffer1[3] == '1') && (buffer1[3] == '0'))) {

It was the presence of buffer1[3] that caught my eye.

I tend to always put brackets in place as I've seen too many people fall foul of the precedence or operators.

In your code what is the definition and value of broadcast?

yes, the buffer[3] is just a type ))
I've added the last digit check for this post, so it's more clear... and made a typo. :slight_smile:
In reality I just check three first digits (because encoder skips, when moving fast).
Here is the real line
if (!reached && buffer1[1] == '4' && buffer1[2] == '8' && buffer1[3] == '1') {
}
The broadcast address is this one:

IPAddress broadcast(192, 168, 1, 255);

15

One thing I have not asked is whether the delay applies to all packets or just the first? If it is just the first then it could be down to MAC address resolution.

When I send packets continuously the delay is also there... :frowning:

I think installing wireshark on either the RPi or the MAC might help to determine whether the packets appear on the RPi (MAC) soon after arriving or are delayed.

hmm... ok. just discovered it's existence... let me try to figure out how it works.
thanks!

Nothing we have noted appears to clarify where the delay is. All we seem to have clarified is that whether you use an Arduino with Ethernet shield or a hardware RS232 to Ethernet converter the RPi (MAC) appears to get it 40s late.

that's correct.
also we sort of know that there is no problem on serial end...

here is the full sketch btw:

#include <Ethernet.h>
#include <EthernetUdp.h>

byte mac[] = {
0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED
};
IPAddress ip(192, 168, 1, 7);
IPAddress broadcast(192, 168, 1, 255);

int length1;
byte buffer1[100];

int length2;
byte buffer2[100];

byte outbuffer[100];
unsigned long lastSentTime = millis();
int counter=0;

unsigned int localPort = 8888; // local port to listen on
EthernetUDP Udp;

void setup() {
Ethernet.init(10); // Most Arduino shields
Ethernet.begin(mac, ip);

Serial1.begin(115200); // read encoder1
Serial2.begin(115200); // read encoder1

if (Ethernet.hardwareStatus() == EthernetNoHardware) {
while (true) {
delay(1); // do nothing, no point running without Ethernet hardware
}
}
if (Ethernet.linkStatus() == LinkOFF) {
Serial.println("Ethernet cable is not connected.");
}

// start UDP
Udp.begin(localPort);
outbuffer[0] = '\0';

pinMode(LED_BUILTIN, OUTPUT);
digitalWrite(LED_BUILTIN, LOW);
}
bool reached = false;

void loop() {

length1 = Serial1.readBytesUntil('-', buffer1,100);
buffer1[length1]='\0';

length2 = Serial2.readBytesUntil('-', buffer2,100);
buffer2[length2]='\0';

if (length1+length2 > 8) {
sprintf(outbuffer,"%s:%s-",buffer1,buffer2);
Udp.beginPacket(broadcast, localPort);
Udp.write(outbuffer, length1+length2+2);
Udp.endPacket();
}
}

and here is video where you can see udp encoder output

thanks!
M

16

here are the wireshark results

wireshark_001.png
wireshark_001.png1680×1050 168 KB

wireshark_001.png
wireshark_001.png1680×1050 168 KB

17

Can we relate the times in wireshark to the time the arduino sends the packet? Are the Arduino & RPi synchonised at all (via NTP?). Could we get an accurate time in the terminal window, or another window when the first data is displayed?

I am wondering if the terminal app is the cause of the delay? Can we use anything else easily?

18

Can we relate the times in wireshark to the time the arduino sends the packet? Are the Arduino & RPi synchonised at all (via NTP?). Could we get an accurate time in the terminal window, or another window when the first data is displayed?

no, they are not synchronised at all right now. I didn't know about NTP, but can try to setup that.
the time in terminal window is certianly possible.

I am wondering if the terminal app is the cause of the delay? Can we use anything else easily?

well, it's unix terminal... I seriously doubt that it causes delay (on Mac and especially on RPi)...

thanks
M

19

Could you read a digital pin on the RPi easily?

Just thinking you turn an LED on using a digital pin on the Arduino, could you also turn on another pin at the same that the RPi could read to give a transmit time reading on the RPi? Not sure how you would this in physical terms as the RPi is 3.3v I believe and the Arduino is 5V - and not wanting to blow the RPi up. Maybe a pair of resistors to split the voltage 3.3 to RPi and 1.7 to ground?

That should allow the determination of where the delay is, at the Arduino end, the network itself or the RPi/Mac end. Seems unlikely the Arduino since the hardware solution you tried shows the same delay. Could be the network having to resolve the MAC address before starting, but then I would expect that should only show on the first packet.

20

countrypaul:
Could you read a digital pin on the RPi easily?

Just thinking you turn an LED on using a digital pin on the Arduino, could you also turn on another pin at the same that the RPi could read to give a transmit time reading on the RPi? Not sure how you would this in physical terms as the RPi is 3.3v I believe and the Arduino is 5V - and not wanting to blow the RPi up. Maybe a pair of resistors to split the voltage 3.3 to RPi and 1.7 to ground?

That should allow the determination of where the delay is, at the Arduino end, the network itself or the RPi/Mac end. Seems unlikely the Arduino since the hardware solution you tried shows the same delay. Could be the network having to resolve the MAC address before starting, but then I would expect that should only show on the first packet.

Hello
Ok, so I did connect Arduino and RPI Gpio together as you suggested.
I did two tests.

  1. I send a signal via digital pins Arduino-RPi right after sending UDP out (when hardcoded encoder value has been reached)

here is the video

as you can see there is 3ms difference between the two (GPIO vs UDP). So latency is very-very small actually for UDP.

  1. I added a LED to RPi, which gets turned on, when RPi recieves GPIO signal from Arduino. (in addition to Arduino turning on it's own LED)

here is the video

again there is no delay, both LEDs light up instantly. However the output from UDP is clearly delayed.

It looks like after all you were right. This is all coming not from UDP, but terminals and all other crap that's inbetween....

Oh well. I think at this point nothing else can be done. I just need to come with some software fixes to compensate for these delays.

thanks,
M