Checking UDP messages are being sent

Hi All,

I’m making a datalogging application using an Arduino Mega 2560 and Ethernet Shield.

I was sending all my values to a webpage and then parsing the webpage using Visual Basic on the PC but it’s too slow.

I’ve decided that I should be using UDP to send the values from Arduino instead.

I’m just experimenting at the moment and so far I know I can send values from the PC to the arduino over UDP.

I can’t seem to receive values from the arduino on the PC over UDP.

Now because I’m writing 2 programs I don’t know where the problem lies so need some help.

here’s my Arduino code:

#include <SPI.h>         // needed for Arduino versions later than 0018
#include <Ethernet2.h>
#include <EthernetUdp2.h>         // UDP library from: bjoern@cs.stanford.edu 12/30/2008


// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = { 0x90, 0xA2, 0xDA, 0x10, 0x05, 0x9B };
IPAddress ip(192,168,2,15); //<<< ENTER YOUR IP ADDRESS HERE!!!

unsigned int localPort = 8888; //Assign a Port to talk over
char packetBuffer[UDP_TX_PACKET_MAX_SIZE];
String datReq; //String for our data
int packetSize; //Size of Packet
EthernetUDP Udp; //Define UDP Object
 
void setup() {
  
Serial.begin(9600); //Turn on Serial Port
Ethernet.begin(mac, ip); //Initialize Ethernet
Udp.begin(localPort); //Initialize Udp
delay(1500); //delay
}
 
void loop() {
  
  packetSize = Udp.parsePacket(); //Read theh packetSize
  
  if(packetSize>0){ //Check to see if a request is present
  
  Udp.read(packetBuffer, UDP_TX_PACKET_MAX_SIZE); //Reading the data request on the Udp
  String datReq(packetBuffer); //Convert packetBuffer array to string datReq
  
  if (datReq =="Red") { //See if Red was requested
  
    Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());  //Initialize Packet send
    Udp.print("You are Asking for Red"); //Send string back to client
    Serial.println("Sending Worked");
    Udp.endPacket(); //Packet has been sent
    delay(1000);
  }
   if (datReq =="Green") { //See if Green was requested
  
    Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());  //Initialize Packet send
    Udp.print("You are Asking for Green"); //Send string back to client
    Udp.endPacket(); //Packet has been sent
   }
    if (datReq =="Blue") { //See if Red was requested
  
    Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());  //Initialize Packet send
    Udp.print("You are Asking for Blue"); //Send string back to client
    Udp.endPacket(); //Packet has been sent
    }
  }
  memset(packetBuffer, 0, UDP_TX_PACKET_MAX_SIZE);
}

I’ve adapted someone else’s code so the only thing i’ve changed is to add a Serial.println(“Sending Worked”); command.

So when I send “Red” over UDP using visual basic to 192.168.2.15 (Arduino’s IP) and port 8888 it executes that line and prints sending Worked over the serial monitor.

What I don’t get is “You are Asking for Red” in my receive textbox in visual basic (I can receive data using this method locally so I know the receive code works locally).

How do I test if the message is being sent? Or am I missing something important?

I've decided that I should be using UDP to send the values from Arduino instead.

Why do you think that is going to be faster?

How do I test if the message is being sent?

Wireshark? TCPviewer? A lot depends on your operating system and skills.

OK, I've got it working now,

I needed to change:

Udp.beginPacket(Udp.remoteIP(), Udp.remotePort());

To:

Udp.beginPacket(Udp.remoteIP(), 8888);

PaulS, I've no idea why UDP would be faster but I figured connecting to the webpage was the slow part of my application but since experimenting with UDP I've found it's not much faster (about 3 seconds rather than 4).

The code I posted was just a test thing but my real code is more complex

I've discovered that the print statements are the slow part as I have 324 of them.

Why so many? Well I'm trying to discover "uptime" or "downtime" of different machines and I want to use as many pins as possible.

So each Pin I have: UpTime, Last UpTime Since the state changed, Unique ID for the last UpTime Since the state changed and DownTime, Last DownTime Since the state changed, Unique ID for the last DownTime Since the state changed.

So each pin has 6 variables that COULD update every second (although not all of them at once as uptime can't increase if downtime is increasing etc...)

54 inputs times 6 variables = 324.

324 print statements over UDP takes about 3 seconds.

So I'm stuck now as I need to print all 324 variables but I need to update every second.

Any Ideas how I can print faster, can I store all my variables into one big variable and send that and somehow parse it the other end in Visual Basic?

54 inputs times 6 variables = 324.

54 because the Mega has 54 pins? $ of them are used for SPI (communication with the Ethernet shield). Two are used by the HardwareSerial instance, Serial.

So, that leaves only 48 pins for your use.

So I'm stuck now as I need to print all 324 variables

No. You only need to print the ones that changed.

Any Ideas how I can print faster

Serial.print() or Client.print()? Part of the issue may be the time it takes to convert the data to strings. You might be able to send binary data and skip that step.

megamef: PaulS, I've no idea why UDP would be faster but I figured connecting to the webpage was the slow part of my application but since experimenting with UDP I've found it's not much faster (about 3 seconds rather than 4).

TCP is a compelled, error corrected, transaction based, protocol. UDP, isn't! In a nutshell, everything else being equal, TCP introduces additional latencies into the conversation.

I've discovered that the print statements are the slow part as I have 324 of them.

If each print statement introduces an additional handshake, that would be a lot of unnecessary latency.

Any Ideas how I can print faster, can I store all my variables into one big variable and send that and somehow parse it the other end in Visual Basic?

You might find these links on my knowledge base useful.

Arduino Device Control Block example using UDP

The decode is in Processing (Java) but it should be easy enough to port to VB

An open TCP/IP connection is plenty fast - its the delay introduced by closing and re-opening the TCP socket that will hit performance, and some HTTP header processing.

Normally you would specify HTTP 1.1 and not use "Connection: close" in the headers if you want the HTTP stream to stay open, which is the way to get swift web access. Generating and processing headers on the Arduino side of things is going to be the next limiting factor.

Using UDP means you have to implement acks and retries and weed out repeated and out-of-order packets as UDP is unreliable and provides no guarantee of one-to-one match up with what's been sent. TCP handles all this and more. Use TCP unless you want to implement lots of networking code.

You might compare the below arduino server code as compared to your original method of data collection. The “fast data” is reasonably fast considering the data is a returned web page. You might want to consider have the pc program request data from the arduino and then do the logging when the data is returned.

// zoomkat's meta refresh data frame test page 5/25/13
// use http://192.168.1.102:84 in your brouser for main page
// http://192.168.1.102:84/data static data page
// http://192.168.1.102:84/datastart meta refresh data page
// for use with W5100 based ethernet shields
// set the refresh rate to 0 for fastest update
// use STOP for single data updates

#include <SPI.h>
#include <Ethernet.h>

const int analogInPin0 = A0;
const int analogInPin1 = A1;
const int analogInPin2 = A2;
const int analogInPin3 = A3;
const int analogInPin4 = A4;
const int analogInPin5 = A5;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // arduino ip in lan
byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
unsigned long int x=0; //set refresh counter to 0
String readString; 

//////////////////////

void setup(){
  Serial.begin(9600);
    // disable SD SPI if memory card in the uSD slot
  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  server.begin();
  Serial.println("meta refresh data frame test 5/25/13"); // so I can keep track of what is loaded
}

void loop(){
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
         if (readString.length() < 100) {
          readString += c; 
         } 
        //check if HTTP request has ended
        if (c == '\n') {

          //check get atring received
          Serial.println(readString);

          //output HTML data header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();

          //generate data page
          if(readString.indexOf("data") >0) {  //checks for "data" page
            x=x+1; //page upload counter
            client.print("<HTML><HEAD>");
            //meta-refresh page every 1 seconds if "datastart" page
            if(readString.indexOf("datastart") >0) client.print("<meta http-equiv='refresh' content='1'>"); 
            //meta-refresh 0 for fast data
            if(readString.indexOf("datafast") >0) client.print("<meta http-equiv='refresh' content='0'>"); 
            client.print("<title>Zoomkat's meta-refresh test</title></head><BODY>
");
            client.print("page refresh number: ");
            client.print(x); //current refresh count
            client.print("

");
            
              //output the value of each analog input pin
            client.print("analog input0 is: ");
            client.print(analogRead(analogInPin0));
            
            client.print("
analog input1 is: ");
            client.print(analogRead(analogInPin1));
                        
            client.print("
analog input2 is: ");
            client.print(analogRead(analogInPin2));
            
            client.print("
analog input3 is: ");
            client.print(analogRead(analogInPin3));
                                    
            client.print("
analog input4 is: ");
            client.print(analogRead(analogInPin4));
            
            client.print("
analog input5 is: ");
            client.print(analogRead(analogInPin5));
            client.println("
</BODY></HTML>");
           }
          //generate main page with iframe
          else
          {
            client.print("<HTML><HEAD><TITLE>Zoomkat's frame refresh test</TITLE></HEAD>");
            client.print("Zoomkat's Arduino frame meta refresh test 5/25/13");
            client.print("

Arduino analog input data frame:
");
            client.print("&nbsp;&nbsp;<a href='http://192.168.1.102:84/datastart' target='DataBox' title=''yy''>META-REFRESH</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/data' target='DataBox' title=''xx''>SINGLE-STOP</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/datafast' target='DataBox' title=''zz''>FAST-DATA</a>
");
            client.print("<iframe src='http://192.168.1.102:84/data' width='350' height='250' name='DataBox'>");
            client.print("</iframe>
</HTML>");
          }
          delay(1);
          //stopping client
          client.stop();
          //clearing string for next read
          readString="";
        }
      }
    }
  }
}

Thanks Guys you’ve helped me a lot.

So I’m getting values on the PC from the Arduino over UDP in about 0.5 seconds now which is fine.

But after everyone’s advice I think I might go back to generating a webpage as the unreliability of UDP is concerning.

Zoomkat, I tried your code out it worked great and and seeing the “fast data” part proved to me that HTTP is more than up to the job so I’m going down that road now.

If you don’t mind could you explain to me what does the timing? The only part that seemed to affect timing was

client.print("<meta http-equiv='refresh' content='1'>");

I found changing the 1 to a 2 changed the timing from 1 second to 2.

Is the Arduino counting to a second and then generating a new webpage?

This is confusing as it’s within a print statement so how does the Arduino “know” when to update?

I need to know as I want my Arduino program to count every second and increase a Downtime/Uptime variable

Is the Arduino counting to a second and then generating a new webpage?

No, the browser is doing the time counting and then making the request. The arduino just returns the page when requested by the browser.

MarkT: An open TCP/IP connection is plenty fast - its the delay introduced by closing and re-opening the TCP socket that will hit performance, and some HTTP header processing.

You are confusing HTTP with TCP. HTTP is an additional layer of protocol. The relationship between HTTP and TCP is like the relationship between fax and a phone line.

When A TCP packet is successfully transmitted and arrives first time, it is a little slower than UDP. When a TCP packet needs to be retransmitted, reordered or delivery is deferred, TCP is hugely slower.

Using UDP means you have to implement acks and retries and weed out repeated and out-of-order packets as UDP is unreliable and provides no guarantee of one-to-one match up with what's been sent. TCP handles all this and more. Use TCP unless you want to implement lots of networking code.

Just, no!

To illustrate.

Imagine you have an Arduino sampling a sensor and sending the data, to a monitoring application across the network, in a loop. After five loops the Arduino has sent values 1, 2, 3, 4, 5. However, a glitch occurs on the network while the packet containing '2' is in transit. Using TCP, the packet is resent and the monitoring application receives 1, 3, 4, 2, 5. Using UDP the packet is dropped and the monitoring application receives 1, 3, 4, 5. Which is more accurate?

When we infer the passing of time at both ends of the network. With UDP the 1, 3, 4, 5 were all received a few milliseconds after they were transmitted. With TCP, the 1, 3, 4, 5 turned up at roughly the same time (as with UDP) but the 2 turned up several seconds later than it was sent. So, the non-compelled UDP stream, turns out to be more accurate.

TCP is transaction based and that (alone) makes it unsuited to many real time applications.

megamef:
But after everyone’s advice I think I might go back to generating a webpage as the unreliability of UDP is concerning.

Some advice is more equal than others(!)

Don’t get hung up on UDP being ‘unreliable’ as it’s all relative. Beyond the pages of text books, on a switched LAN, for sending small packets of data (< 1KB), UDP proves to be every bit as reliable as TCP.

HTTP requires hundreds of bytes of additional protocol. The HTTP protocol is vague by design, which leads to all sorts of ‘edge’ cases. I would think seriously hard, whether the complication of HTTP will benefit the application.

Thanks again guys,

I've taken on board what everyone has said and reduced my number of pins so I'm now using only pins 22 - 49 this means I'm only using the bottom row of the mega so makes connection later on simple and I avoid ChipSelect and SPI pins, also I'm only sending data when I need to. All this has drastically reduced the time it take to send the data.

I've got both HTTP and UDP methods working in the same project so that they are triggered to send data only when I press a button on my visual basic program.

I can clearly see that UDP is quicker, it's about 0.5 seconds vs about 1.0 second for HTTP.

I'm still of two minds which to use. I might use HTTP because I'm generating a webpage so I can check the status of the output independent of my VB program, but at the same time UDP is quicker and i want the end user to be able to see data without delay.

Might even use both: UDP when a user requests data (as it's quickest) but also auto call HTTP every min or so to save the latest data to my database.

If your users can't wait the extra half second, shame on them.

megamef: I'm still of two minds which to use. I might use HTTP because I'm generating a webpage so I can check the status of the output independent of my VB program, but at the same time UDP is quicker and i want the end user to be able to see data without delay.

i) How about a simple loop, sending simple data in UDP packets, from the micro-controller. ii) Your VB code is a simple UDP listener which updates a database. iii) A second process on the PC queries the database and renders web-pages on request. iv) Maybe this 2nd PC process is IIS, Apache or NGinx and uses ASP or PHP to render the web page.

PaulS: If your users can't wait the extra half second, shame on them.

Then shame on me. I send and receive 10 UDP packets per second. I can't wait a half second. My code also returns an ack packet from the UDP "server", just like NTP.

BTW, I think UDP is much faster and easier to manage than TCP. It is as fast as establishing the TCP connection.