MKR ETH Speed is slow

Folks,

I'm using an MKR ZERO plus an MKR ETH to repeatedly send 128 bytes to an IP address. There is no software listening at that destination address, but I don't think that should affect a UDP transmission.

The destination computer is directly connected to the MKR (no intervening switch or other network equipment, other than a cable), but again, I don't think UDP cares.

I'm using the 1.8.9 Arduino IDE to compile, and I'm using the 2.0.0 version of the built-in Ethernet library.
Arduino/Reference/Ethernet

In the MKR ZERO, the EthernetUDP.endPacket() invocation for each of my 128 byte transmissions (all zeros) is taking 1.6 seconds to complete.

That's absurdly long/slow: ( 128 * 8 ) bits / 1.6 seconds = 640 bits per second.

I'm curious if anyone can spot a mistake in my simple program, or can describe to me why a short message would take so long to send (surely there isn't that much overhead (> 1 second) per message).

Here's the code, and some sample results. In the results, the numbers are the times spent executing the endPacket() invocations, in milliseconds.

I have my fingers crossed that the answer is that I'm doing something boneheaded, or that there are some simple SPI or other tricks for speeding things up.

#include "Ethernet.h"
#include "EthernetUdp.h"
#include "IPAddress.h"

//=======================================================================================//
//============================================================================//

EthernetUDP _udp;
byte        _myMacAddress[6] = {0x01, 0x02, 0x03, 0x04, 0x05, 0x06};
IPAddress   _myIpAddress     = IPAddress{192, 168, 192, 223};;
int         _myPort          = 14531;

IPAddress   _heartbeatIp     = IPAddress(192, 168, 192, 232);
int         _heartbeatPort   = 17142;

byte        _outboundPacketBuffer[128];

//=======================================================================================//
//=======================================================================================//

void setup()
{
  Serial.begin(9600);
  delay(5000);

  Serial.println("\nDoing UDP Setup");

  Serial.print("   My MAC:  "); printHex(_myMacAddress, 6);
  Serial.print("   My IP:   "); printDottedDecIp(_myIpAddress);
  Serial.print("   My Port: "); Serial.println(_myPort);
  
  Serial.println("The default External Computer (laptop) address is");
  Serial.print("   Ext IP:   "); printDottedDecIp(_heartbeatIp);
  Serial.print("   Ext Port: "); Serial.println(_heartbeatPort);
  
  Serial.print("MAX OUTBOUND MESSAGE BUFFER length = "); Serial.print(sizeof(_outboundPacketBuffer)); Serial.println(" bytes\n");

    Ethernet.begin(_myMacAddress, _myIpAddress);
  _udp.begin(_myPort);

  if (Ethernet.hardwareStatus() == EthernetW5500)
    Serial.println("W5500 Ethernet controller detected.\n");
  
  if (Ethernet.linkStatus() == LinkON)
    Serial.println("Link status: ON.  This implies an Ethernet port exists and a useable cable is plugged into it\n");
 }

void loop()
{
  memset(_outboundPacketBuffer, 0, sizeof(_outboundPacketBuffer));

  _udp.beginPacket(_heartbeatIp, _heartbeatPort);  // The remoteIp and remotePort calls only work after doing a parsePacket call
  _udp.write(_outboundPacketBuffer, sizeof(_outboundPacketBuffer));
  
  unsigned long heartStartMicros = micros();
  _udp.endPacket();  // This actually causes the packet to be sent
  double heartSendingMillis = 0.001 * (micros() - heartStartMicros);
  
  Serial.println(heartSendingMillis);
  
  delay(3000);
}

//=======================================================================================//
//     MISC UTILITIES
//=======================================================================================//

void printHex(const byte * data, const uint32_t numBytes)
{
  uint32_t szPos;
  for (szPos = 0; szPos < numBytes; szPos++)
  {
    //    Serial.print("0x");
    // Append leading 0 for small values
    if (data[szPos] < 0x10)
      Serial.print("0");
    Serial.print(data[szPos], HEX);
    if ((numBytes > 1) && (szPos != numBytes - 1))
    {
      Serial.print(' ');
    }
  }
  Serial.println();
}

void printDottedDecIp(uint32_t ip)  // Expects the IP Address to be stored with the last Octet in the MSByte.
{
  Serial.print  ((byte)(ip >>  0)); Serial.print('.');
  Serial.print  ((byte)(ip >>  8)); Serial.print('.');
  Serial.print  ((byte)(ip >> 16)); Serial.print('.');
  Serial.println((byte)(ip >> 24));
}

//=======================================================================================//
//=======================================================================================//
Doing UDP Setup
   My MAC:  01 02 03 04 05 06
   My IP:   192.168.192.223
   My Port: 14531
The default External Computer (laptop) address is
   Ext IP:   192.168.192.232
   Ext Port: 17142
MAX OUTBOUND MESSAGE BUFFER length = 128 bytes

W5500 Ethernet controller detected.

Link status: ON.  This implies an Ethernet port exists and a usable cable is plugged into it

1617.90
1617.76
1617.77
1617.67
1617.76
1617.62
1617.80
1617.65

Have you tried the same code on a different Arduino? Two data points make for more sense than one, when trying to understand where a problem is happening.

Thanks Paul

Yes, other boards are behaving similarly - I just repeated the test using three other MKR ZERO plus MKR ETH board pairs.

One of those three pairs was 100% brand-new fresh-out-of-the-box.

Same results for all three new/different pairs, times slightly over 1.6 seconds.

Nothing other than a USB cable and an Ethernet cable is connected to these boards.

I don't want to get lost down a rabbit hole, unless it's the best course of action (I suppose it depends on whether my current results are easily duplicated by someone else), but I might have been seeing much faster transmissions a few weeks ago. If I could be sure of that I would have written it above. However, if my speeds were better a while ago, then any recent library, etc. upgrades would be suspect.

I suppose my first curiosity is whether someone with the same equipment I'm using duplicates, or doesn't, what I'm seeing. If someone else gets different results, we can start figuring out why.

is the message received on the computer?

Thanks Juraj,

Even though I don't think I care whether the message gets to a destination, your question did properly motivate me to check the return codes reported by the three EthernetUDP subroutine calls.

beginPacket() returns 1 = success
write() returns 128 = number of bytes "sent"
endPacket() returns 0 = failure

I wonder how many different problems result in endPacket() returning 0? It sure would be handy to have a list of them right now.

Does anyone know where to look for that list of reasons when endPacket might report a failure (please don't say "The source code")?

FYI: Even this simple tweak of the EthernetUDP code from the Arduino Reference Pages (Reference/EthernetUDPEndPacket) is exhibiting the same symptom.

#include <SPI.h>        
#include <Ethernet.h>
#include <EthernetUdp.h>

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = {  
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
IPAddress ip(192, 168, 1, 177);

unsigned int localPort = 8888;      // local port to listen on

// An EthernetUDP instance to let us send and receive packets over UDP
EthernetUDP Udp;

void setup() 
{
  // start the Ethernet and UDP:
  Ethernet.begin(mac,ip);
  Udp.begin(localPort);
}

int _count = 0;
void loop() {

    IPAddress destinationIp(192, 168, 1, 178);
    int destinationPort = 8889;
    int rc1 = Udp.beginPacket(destinationIp, destinationPort);
    int rc2 = Udp.write("hello");
    int rc3 = Udp.endPacket();

    Serial.print(_count++);
    Serial.print(", ");
    Serial.print(rc1);
    Serial.print(", ");
    Serial.print(rc2);
    Serial.print(", ");
    Serial.println(rc3);
    delay(3000);
}

if Ethernet has a problem on lower level like if it can't resolve IP of target to MAC of target, it matters for UDP too

Juraj wins the prize for helping me remember that it's been a very long time (apparently too long) since I studied how computers, routers, and switches, use the PHY/IP/UDP/TCP/ARP/RARP/ICMP/etc. protocols to interact with one another.

With the MKR transmitting into a switch to send a message to another computer (on the same switch) that has the IP address the MKR is transmitting to, EndPacket() is taking about 62 micros

Unplugging the destination computer created a +/- 2 micros transient, and then the time settled back down to 62 micros.

Rebooting the MKR while the destination computer remains unplugged from the switch, drives the MKR EndPacket() time up to 1.6 seconds.

Plugging the destination computer into the switch, drops the EndPacket() time back down to 62 micros.

According to one source on the web (I don't vouch for the source' correctness):
"[when you try send a] UDP packet to an IP address that is not known by your machine, [your machine] will ask for the [other] machine's MAC address first via the ARP protocol. If it gets a response, it will send your packet to the MAC address it receives, if it cannot get a response about the MAC address, the UDP packet won't be sent at all."

So, I assume that:

  • When/if EthernetUDP gets a destination MAC address, EthernetUDP caches it, and uses it forever (the 62 microsecond EndPacket() executions); and
  • When EthernetUDP doesn't have a MAC address for the destination (and that MAC address can't be gotten from any source), EthernetUDP spends some non-trivial time trying to learn the destination's MAC, before eventually giving up and failing to send the message (the 1.6 second EndPacket() executions).
    I'm ready to move on to my next task, but if anyone wants me to do another experiment or two, post a request here (soon-ish), and I'll try to accommodate you.

Thanks again Juraj.

The 1600 ms timeout seems ridiculously conservative. I would much prefer it to fail quickly so I can run whatever other code I need to and then try again after an interval. I found that you can adjust the timeout using Ethernet.setRetransmissionCount(). Note that the argument to setRetransmissionCount specifies the total number of transmission attempts so the minimum value is 1. By default, it makes 8 attempts, each of which has a 200 ms timeout, thus the 1600 ms. With TCP/IP, you can set the timeout duration via Ethernet.setRetransmissionTimeout() but for some reason this has no effect with UDP.

Thanks for the info Paul,

And... Color me surprised.

Because they are, by definition, "unreliable", I never would have guessed (or recalled) that a UDP (or IP) implementation would delay execution by retrying anything affecting this subject, or that altering a RetransmissionCount would affect that behavior.

I definitely need to invest some of my copious spare time :roll_eyes: in refreshing my recollection of the appropriate protocol specs/descriptions.

PS: Did you happen to spot a public routine for lowering the pertinent timeout value? In my simple not-connected-to-anything-else LAN, 200ms is a long time to wait for something that probably will either take microseconds, or not ever succeed until much later.

PS: From here, Timeout-delay-in-sendUDP-in-endPacket-when-target-is-not-online I see that in July of 2018, one implementation of W5100 code included a "W5100.setRetransmissionTime(#)" function, for which I think # represents an integer number of milliseconds.

PPS & FYI: I've done a little looking. I don't see a single hint in any of the references I've checked, that retrying an ARP address request is either expected or desirable, or that a blocking timeout should be associated with waiting for a reply to one.

In fact the opposite seems to have been the case for a long time (Maybe there's a new RFC?). This is wrapped up in "First UDP Message gets Lost" sorts of reports from UDP users that send long messages.

If anyone reading this gets a chance to poke the WizNet and/or EthernetUDP folks about the subject, maybe they should stop defaulting to using a non-zero timeout, and stop defaulting to using a non-zero retry count (Paul, are you an EthernetUDP author?).

PPPS: In another post above, I speculated that EthernetUDP (or the WizNet code?) might keep an ARP table entry forever, once the entry is inserted into its table. Many (most?) implementations don't keep entries forever, and will delete them if they become stale (20 minutes is one suggested lifetime for a table entry). I don't know how the EthernetUDP/WizNet combination handles this.

IF an otherwise valid entry is deleted by an ARP implementation, because that entry is old, the transmitter then will need to go through the process of re-learning that entry.

The time needed to carry out that relearning process will create a little hiccup in the time need to complete the affected transmission (in other words, occasionally, for no immediately obvious reason, a transmission will take longer than is typical).

PPPPS: I don't mind being the millionth person to write this, "Wouldn't it be nice if the Arduino Reference web pages contained this sort of info, or contained a link to an official source for the info? I would spend a few extra coins per board to pay the salary of the person who would create and maintain that library."

Metron_Ross:
Thanks for the info Paul

Since it seems to be a response to my previous reply, I'll go ahead and answer to this. Of course anyone named Paul is also welcome.

Metron_Ross:
Because they are, by definition, "unreliable", I never would have guessed (or recalled) that a UDP (or IP) implementation would delay execution by retrying anything affecting this subject, or that altering a RetransmissionCount would affect that behavior.

I didn't know about this before now either. I actually have almost no experience with UDP and the Ethernet library. That 1600 ms you got seemed like too much of a coincidence, so I did the experiment with Ethernet.setRetransmissionCount() and Ethernet.setRetransmissionTimeout() and found the former had the effect of changing the delay by 200 ms per count. I didn't dig any further than that.\

Metron_Ross:
one implementation of W5100 code included a "W5100.setRetransmissionTime(#)" function, for which I think # represents an integer number of milliseconds.

Ethernet.setRetransmissionTimeout() is just calling W5100.setRetransmissionTime(). Ethernet.setRetransmissionTimeout() was added in the last release of the Ethernet library. I think the idea is to provide a more standardized implementation that could be used across all versions of the Ethernet library (for example, the Intel x86 version of the Ethernet library doesn't have W5100.setRetransmissionTime()) and maybe to make this incredibly useful functionality more accessible to the average Arduino user. After all, the "W5100" part is a bit confusing when you're using a W5500.

I also did a test of your code with W5100.setRetransmissionTime() and it still had no effect on the delay. Note that the argument to Ethernet.setRetransmissionTimeout() is milliseconds, while with W5100.setRetransmissionTime() it's 0.1 ms.

Metron_Ross:
Paul, are you an EthernetUDP author?

I actually was indirectly the inspiration behind the addition of the Ethernet.setRetransmissionTimeout() and Ethernet.setRetransmissionCount() functions. I published a modified version of the Ethernet library that added the automated W5x00 chipset detection added years later in the official Ethernet 2.0.0 release. I didn't actually write that code, I only copied it from a pull request that had been submitted to Arduino years earlier but was never merged for some reason. I did the same with the EthernetClient remoteIP and remotePort functions. I did write the EthernetClient setConnectionTimeout function and documented the use of W5100.setRetransmissionTime() and W5100.setRetransmissionCount(). When Paul Stoffregen did the big update of the Ethernet library to add the automated W5x00 chipset detection, they came across some references to my library and incorporated all my changes into the official Ethernet library, even going to the additional step to add the wrappers for W5100.setRetransmissionTime() and W5100.setRetransmissionCount(). Beyond that, my only involvement in the development of the Ethernet library was writing the documentation for all the new functions added in the 2.0.0 release and a bunch of little documentation fixes over the years before that. I am one of the maintainers of the Ethernet library repository, but only to manage the issue tracker.

Metron_Ross:
Wouldn't it be nice if the Arduino Reference web pages contained this sort of info, or contained a link to an official source for the info?

It's not clear to me exactly which info you're referring to. I think it would be good to document the information we have just found about the effect of Ethernet.setRetransmissionCount() and the non-effect of Ethernet.setRetransmissionTimeout() on UDP, since especially the latter is non-obvious. I do think that we need to be careful to keep the documentation beginner friendly. That means providing all the information they need, but also not adding a bunch of technical details they don't need. I'm in favor of adding links to external resources where the curious can learn more.

If you have specific suggestions for improving the documentation, please open an issue report on GitHub. I think the Ethernet library repository is the most appropriate place to open issues about the Ethernet documentation:

I'll be honest with you and tell you that it could be a very long time before any action is taken on those issues. If I had my druthers, I'd act on any non-controversial suggestions immediately, but I don't have edit access to that content. All my contributions to the Ethernet documentation have been accomplished by opening issue reports on GitHub so the system does work occasionally.

If you have suggestions for improvements to the Arduino Language Reference content, the situation is different. There is a nice system where the content is all hosted in a GitHub repository, from which the arduino.cc content is automatically updated. I am a maintainer of that repository so if you submit a pull request to make a change to that content, I can merge your pull request, and I try to be very active in my management of that repository. If you submit an issue report to the repository, then I will submit a pull request to resolve the issue if possible, but I'm not allowed to merge my own pull requests so then we have to wait for one of the other less active maintainers to merge it.

Pert,

Thanks.

And, Yes, I did confuse you with Paul. I thought your reply was a second comment from him.

About the code and documentation.

  • For me, beginner friendly code and documentation would mean that the Arduino MKR UDP (and ARP, and ...) protocol stack would behave the way the UDP protocol stacks generally behave on most other computers, and that any behavior that differs from what is typical for other computers would be clearly described.

  • I'm not going to claim to know for certain what's typical for other protocol stacks, but I am confident enough to risk betting a nice dinner that UDP messages destined for an address currently not known by the sender's ARP implementation, usually trigger one ARP transmission (Not six), and that UDP senders aren't held up waiting for ARP/UDP timeouts (especially not 200 msec timeouts).
    If what we are talking about here was a technical detail people (beginner or not) didn't need, we wouldn't be talking about it.

Wouldn't you agree that just like everyone else, beginners expect that what they learn about UDP communication elsewhere will apply when they are using libraries promoted on the Arduino Reference pages, and that they also expect to see all of a libraries interfaces listed & described when they look at the Arduino Reference pages.

I may get around to using github's interface to submit some suggestions, but I have no idea when that will be.

Thanks again for your help, and for being a friendly ear.

Metron_Ross:
Wouldn't you agree that just like everyone else, beginners expect that what they learn about UDP communication elsewhere will apply when they are using libraries promoted on the Arduino Reference pages

Yes, unless the library documentation states otherwise. Any intentional differences should be clearly documented in the library reference (and unintentional ones should be fixed).

Metron_Ross:
and that they also expect to see all of a libraries interfaces listed & described when they look at the Arduino Reference pages.

Absolutely. Is there an interface of the Ethernet library that's not documented?

pert:
I found that you can adjust the timeout using Ethernet.setRetransmissionCount(). Note that the argument to setRetransmissionCount specifies the total number of transmission attempts so the minimum value is 1. By default, it makes 8 attempts, each of which has a 200 ms timeout, thus the 1600 ms.

FYI: Unless there is some non-obvious problem lurking undetected, I think it is legal to pass an argument of zero into Ethernet.setRetransmissionCount().

I tried passing an argument of one to the setRetransmissionCount() function. The result was that failing EthernetUDP.endPacket() attempts took about 350 milliseconds.

I tried passing an argument of zero to the setRetransmissionCount() function. The result was that failing EthernetUDP.endPacket() attempts took about 169 milliseconds.

After setting the argument to zero, once the EndPacket() calls started succeeding (because the destination computer became available), the now successful EndPacket() calls took about 60 microseconds.

pert:
Yes, unless the library documentation states otherwise. Any intentional differences should be clearly documented in the library reference (and unintentional ones should be fixed).
Absolutely. Is there an interface of the Ethernet library that's not documented?

Ah ... My mistake - The Ethernet.setRetransmissionCount() is an "Ethernet" api, not an EthernetUDP api, and it is listed in the Ethernet section.

Metron_Ross:
FYI: Unless there is some non-obvious problem lurking undetected, I think it is legal to pass an argument of zero into Ethernet.setRetransmissionCount().

I tried passing an argument of one to the setRetransmissionCount() function. The result was that failing EthernetUDP.endPacket() attempts took about 350 milliseconds.

I tried passing an argument of zero to the setRetransmissionCount() function. The result was that failing EthernetUDP.endPacket() attempts took about 169 milliseconds.

OK, that information is based on my experience with TCP/IP. It's odd that it works differently with UDP. This definitely needs to be documented.