I am trying to communicate with a Ardunio Uno board with a Ethernet Shield. I am running to a issue with trash in the packet. I have traced it down to the data rate at which I am sending the packets to the Ardunio, but I want to ask if anyone else was experiencing this issue.
The packet payload is less than 64bytes, but is a custom protocol. I am using the default UDPSendReceiveString sketch (for testing) with one modification, UDP_TX_PACKET_MAX_SIZE has been increased to 64 from 24, to allow for the planned packet size. I have limited the data rate of the send to about 100ms, 10 packets per second, but I am still getting trash while reading the packet. I have decreased the rate to 200ms, 5 packets per second and the issue does not a occur, but this seems a bit slow for the Ardunio. I did try removing the delay from the end of the routine in the sketch, thinking that this was limiting the Ardunio ability to process the packets are faster rates, but that had no affect on it.
I had already planned to included error checking the communication (of the packet data) and expected some issue with UDP packets, the error is fairly consistent and at the end of the string, but there is packet lost. I have used a packet sniffer to verify that the packets are not malformed/are correct.
Thank you,
Aaron
The example below has two different packet sizes of 44 and 46 bytes. Errors/mistakes are bold.
Received packet of size 44
From 192.168.3.41, port 36769
Contents:
00100000000020000000001000000000400000177999
Received packet of size 46
From 192.168.3.41, port 36769
Contents:
0160000000002000000000300000000040000500006007[b]À¨ )¡y[/b]
Command packet
Received packet of size [b]78[/b]
From [b]48.48.50.48[/b], port [b]12336[/b]
Contents:
00001000000000100000192999[b]À¨ )¡y[/b]
Received packet of size [b]58[/b]
From [b]48.48.50.48[/b], port [b]12336[/b]
Contents:
168999[b]À¨ )¡y[/b]
Received packet of size 44
From 192.168.3.41, port 41337
Contents:
00100000000020000000001000000000400000177999
Received packet of size 46
From 192.168.3.41, port 41337
Contents:
0160000000002000000000300000000040000500006007
The problem is with the implementation of Udp.read(x,x) and Udp.parsePacket();
Udp.parsePacket() does not return the length of the packet, but the length of the receive buffer. Therefore when reading the buffer, it reads past the end of the packet and into the next packet, which is waiting in the buffer. Then when reading the next packet, the read function starts reading some point in the middle of the packet (where it stop while reading the previous packet), therefore corrupting the next packet, too.
I have thought of two work-runarounds for this. 1) Not always possible, but works in my case, use a fixed length packet and read x number of bytes from the receive buffer. Issue: Requires padding the packet and a flushing the entire buffer if a byte is missed, as there would be offset in the buffer, which would corrupt all following packets (until the end of the buffer). 2) Use a special character or some other unique terminator for the packet, and then read/search of all data between the terminators, which would be the packet. Issue: More complicated/complex.
Does anyone have a thought or comment?
Solution: If there was a method to just read the next packet, without specifying the length (aka you don't need to know the length of the expected packet/variable packet length), which would work far better.
I just did a clean install, I just uncompressed the file (arduino-1.0), opened the default UDPSendReceive sketch, changed the IP and port number. I then sent the Arduino Uno ~10 packets rapidly and this is the results.
UDPSendReceive.pde:
created 21 Aug 2010
by Michael Margolis
#include <SPI.h> // needed for Arduino versions later than 0018
#include <Ethernet.h>
#include <EthernetUdp.h> // UDP library from: bjoern@cs.stanford.edu 12/30/2008
Received packet of size 784
From 192.168.3.41, port 59142
Contents:
001000000000200000000010L
Received packet of size 752
From 48.48.48.48, port 12336
Contents:
29876543210999 L
Received packet of size 720
From 32.32.32.32, port 8224
Contents:
À¨ )ç
Received packet of size 688
From 48.48.48.48, port 12336
Contents:
0000000020000000192999 L
Received packet of size 656
From 32.32.32.32, port 8224
Contents:
À¨ )ç
Received packet of size 624
From 48.48.48.48, port 12848
Contents:
000000300000000020000000L
Received packet of size 592
From 49.54.56.57, port 14649
Contents:
À¨ )ç
Received packet of size 560
From 48.48.49.48, port 12336
Contents:
000020000000004000000000L
Received packet of size 528
From 50.48.48.48, port 12336
Contents:
003999 L
Received packet of size 496
From 192.168.3.41, port 59142
Contents:
001000000000200000000050L
Received packet of size 464
From 48.48.48.48, port 12336
Contents:
20000000177999 L
Received packet of size 432
From 32.32.32.32, port 8224
Contents:
À¨ )ç
Received packet of size 400
From 48.48.48.48, port 12336
Contents:
0000000020000500006007 L
Received packet of size 368
From 32.32.32.32, port 8224
Contents:
À¨ )ç
Received packet of size 336
From 48.48.48.48, port 12848
Contents:
000000600000000020000000L
Received packet of size 304
From 49.57.50.57, port 14649
Contents:
À¨ )ç
Received packet of size 272
From 48.48.49.48, port 12336
Contents:
000020000000007000000000L
Received packet of size 240
From 50.48.48.48, port 12336
Contents:
168999 L
Received packet of size 208
From 192.168.3.41, port 59142
Contents:
001000000000200000000080L
Received packet of size 176
From 48.48.48.48, port 12336
Contents:
20000000003999 L
Received packet of size 144
From 32.32.32.32, port 8224
Contents:
À¨ )ç
Received packet of size 112
From 48.48.48.48, port 12336
Contents:
0000000020000000041999 L
Received packet of size 80
From 32.32.32.32, port 8224
Contents:
À¨ )ç
Received packet of size 48
From 48.48.48.48, port 12848
Contents:
000001000000000020000500L
Received packet of size 16
From 48.48.54.48, port 12343
Contents:
20000500L
int EthernetUDP::parsePacket()
{
if (available() > 0)
{
//HACK - hand-parse the UDP packet using TCP recv method
uint8_t tmpBuf[8];
int ret =0;
//read 8 header bytes and get IP and port from it
ret = recv(_sock,tmpBuf,8);
if (ret > 0)
{
_remoteIP = tmpBuf;
_remotePort = tmpBuf[4];
_remotePort = (_remotePort << 8) + tmpBuf[5];
// When we get here, any remaining bytes are the data
ret = available();
}
return ret;
}
// There aren't any packets available
return 0;
}
... that available() returns the original packet size, not the size decreased by 8?
I don't think that is the issue, the problem has to do with the reading of incoming packets. The Arduino sketch does not properly handle the variable length of the incoming packets. If the Arduino does not read the incoming packet before another packet is received, the sketch either truncates the existing packet and does not report the next entire packet.
Received packet of size 11
From 192.168.3.41, port 44876
Contents:
1234567890A
Received packet of size 32
From 192.168.3.41, port 44876
Contents:
1234567890ABCDEF12345678L
Received packet of size 10
From 192.168.3.41, port 44876
Contents:
1234567890ABCDEF12345678L (The real packet contents was "1234567890" not "1234567890ABCDEF12345678L"
I tried adding the following line just prior to the end of the loop of the sketch, but without knowledge of the current packets length (and not the current length of the entire incoming buffer). However, it clears the entire packet buffer.
The use of a constant length packet does allow for reading of the buffer. However, see my previous post.
This is more of a implementation issue/change, the current implementation allows for reading the packet buffer to certain point. But, if it were possible to know/read to the start of the next packet, I think that would fix the issue.
Do things work correctly if the packet size is put back to the original 24 bytes? I'm thinking your max packet size parameter is not fully propagated throughout the ethernet class and you are getting a buffer overrun.
Yes, ajofscott, for my last post, I was using the default UDPSendReceive sketch (other than the port number and IP address).
This is the packets being sent.
First packet:1234567890
Second packet: 1234567890A
Third packet: 1234567890AB
Forth packet: 1234567890ABCDEF1234567890ABCDEF (which is received incorrectly)
Fifth packet: 1234567890 (which is received incorrectly)
Debugging window:
Received packet of size 10
From 192.168.3.41, port 60200
Contents:
1234567890
Received packet of size 11
From 192.168.3.41, port 60200
Contents:
1234567890A
Received packet of size 12
From 192.168.3.41, port 60200
Contents:
1234567890AB
Received packet of size 32
From 192.168.3.41, port 60200
Contents:
1234567890ABCDEF12345678L
Received packet of size 10
From 192.168.3.41, port 60200
Contents:
1234567890ABCDEF12345678L
By my count you recover exactly 24 bytes of the 32 sent. Does it recieve correctly if 24 bytes sent? Send 2 packets with completely different data, it is hard to tell what is happening with exactly the same bytes being sent each time. I want to know if a transmission retry is being appended to the end of the current buffer contents. Your first 32 byte packet arrived up to 24 bytes, then repeated with the subsequent 10 byte packet. Differing data between packet lenths may make it a little eaier to discover what is going on. what are you sending messages from? A winsock control?
UDP is not guaranteed delivered, uncorrupted or generally s**t on by higher level protocols, it is the bitch of TCP/IP and every other protocol on the wire, and can be throttled, stamped on or overwritten at any time.
That is the nature of the beast.
Moving on to the Ethernet libraries, one of the biggest issues you will have with UDP (after ruling out coding problems), is the fact that the Arduino ethernet library is NOT interrupt driven.
If you are playing with UDP and want it to be semi-'reliable' you are at its beck and call, meaning: you better deal with it as soon as it arrives, not when the code feels like it.
Which involves grabbing the interrupt from the ethernet chip, moving the data to a secondary buffer, then returning from the interrupt, finally processing the 'buffer' on a secondary user thread, or writing your own routines that interact DIRECTLY with the ethernet chip and some very 'wing and a prayer' looping code.
UDP is more for a situation where the higher level protocols can allow significant corruption/packet loss.
From the W5100 datasheet:
UDP provides unreliable and connectionless datagram transmission structure. It processes
data without connection establishment. Therefore, UDP message can be lost, overlapped or
reversed. As packets can arrive faster, recipient can not process all of them. In this case, user
application should guarantee the reliability of data transmission.
found this topic when thinked about asking the same here: i'm trying to send a packet with a message format broadcasted to the network, and a server listens it and replyes with a specific message.
OK, it's a known fact that UDP isn't reliable, but is this the behaviour that should be expected?
Grabbing data from a middle of the packet and some from other packet?
My solution now it's to give the servers message a specific format and while that format was not found looping sending requests and parsing packets. It's not very efficient but after a couple of iteractions finally the message is parsed with sucess.
The right way should be by using interrupts to guarantee the packet correct treatment?
Another question: between the send UDP packet and parsepacket to see if there's an packet should exist a delay? Anyone has a proven value for this. I've played around with some values but guess this is a wild guest has it must depends on the networks speed/status and machine involved too.
The best aproach by now seems to looping around.
Has anyone found a better approach to get the packets content parsed?
OK, it's a known fact that UDP isn't reliable, but is this the behaviour that should be expected?
Grabbing data from a middle of the packet and some from other packet?
No it shouldn't do that. However the OP never clarified exactly what the suggested code changes were.
If you do not disable or initialize the SD SPI interface in your setup routine, you will probably get trash in your UDP packets. My NTP program had problems with trash until I disabled that interface.
Catching with this, had manage to get messages correctly.
In the arduino 1.0 examples in order to get a packet we must do:
Udp.read(packetBuffer,UDP_TX_PACKET_MAX_SIZE);
But observing i've seen that packet lenght received in not always 24 as defined in UDP_TX_PACKET_MAX_SIZE.
So, what i've done is simply read the received packetsize , and not that fixed size UDP_TX_PACKET_MAX_SIZE.
Until now works correctly
Udp.read(packetBuffer,packetSize);
Wiresharking the network all packets are correct... so i think because we're reading values bigger/smaller from case to case makes the messages totally garbaged.
Caution to not read more than buffer size is advisable.
edgardurao:
Catching with this, had manage to get messages correctly.
In the arduino 1.0 examples in order to get a packet we must do:
Udp.read(packetBuffer,UDP_TX_PACKET_MAX_SIZE);
But observing i've seen that packet lenght received in not always 24 as defined in UDP_TX_PACKET_MAX_SIZE.
So, what i've done is simply read the received packetsize , and not that fixed size UDP_TX_PACKET_MAX_SIZE.
Until now works correctly
Udp.read(packetBuffer,packetSize);
Wiresharking the network all packets are correct... so i think because we're reading values bigger/smaller from case to case makes the messages totally garbaged.
Caution to not read more than buffer size is advisable.
Does that mean it is working? It is still working here. I'm getting a UDP packet every minute from my NTP server without any trash.
edit: This has been successfully tested on v1.0 and v1.0.1 rc1
For me it's working. Finally i can read the packet from the start. Sometimes there's some garbage in the end of data, but since i know what to read it's easy.
Using the example data is corrupted. Sometimes read data is from the middle of the packet or something more strange.
But it makes sense, reading the size of the package received, and not a fixed value i think. But this must need some validations. It's been a long since i study UDP, and someone can try to see this from a different approach.