Ethernet Shield - Two steps communication protocol

Hi everyone.

I wish to implement a “two step communication protocol” between my arduino board + eth shield and my notebook.

Currently, I’m using arduino as a client and my pc like a server (altough I need to do the opposite as well).
My server is a simple C socket that performs two read() and two write() from/to the client.

I started from the ide’s web client/server examples, but I cannot reach a correct solution to the problem.

Here is the code:

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 33, 17 };  // arduino
byte gateway[] = { 192, 168, 33, 13 };
byte server[] = { 192, 168, 33, 16 };  // notebook

Client client(server, 8001);

void setup() {
  Ethernet.begin(mac, ip, gateway);
  Serial.begin(9600);
  
  // give the Ethernet shield a second to initialize
  delay(1000);
  Serial.println("connecting...");

  if (client.connect()) {
    Serial.println("connected");
    Serial.println("- Sending first message -");
    sendClientMessage("First client message");
    Serial.println("- Reading first message -");
    readServerMessage();
//    readServerMessageWithLen(20);
    Serial.println("delay...");
    delay(500);
    Serial.println("- Sending second message -");
    sendClientMessage("Second client message");
    Serial.println("- Reading second message -");
//    readServerMessageWithLen(21);
    readServerMessage(22);
  } 
  else {
    Serial.println("connection failed");
  }
}

void loop()
{  
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();

    for(;;)
      ;
  }
}

void sendClientMessage(char *msg) {
  client.println(msg);
}

void readServerMessage() {  
  while (client.available()) {
    char c = client.read();
    Serial.print(c);
  }  

void readServerMessageWithLen(int len) {  
  int i = 0;
  while (client.available() && i < len) {
    char c = client.read();
    Serial.print(c);
    i++;
  }  
}

I never get correctly the second message. Here is an example of output:

connecting...
connected
- Sending first message -
- Reading first message -
First server messageSecond server message
delay...
- Sending second message -
- Reading second message -
disconnecting.

The two client messages are received together with the first call to the readServerMessage() function, while the server does not receive data from the second sendServerMessage() call.
I tried also inserting a sleep(1) call between the second read and write of the server, but without success.

I only succeed using the readServerMessageWithLen() function, that stops when the end of the input buffer is reached:

connecting...
connected
- Sending first message -
- Reading first message -
First server message
delay...
- Sending second message -
- Reading second message -
Second server message
disconnecting.

But, obviously, only the server knows about the buffer length…

Am I missing something? :-?

Thanks in advance.

The problem appears to me to be on the server side. The server is not reacting to the specific input. When it receives any input it sends back a whole bunch of data, some of which is relevant, and some of which isn’t.

The client can, then, not determine what is in response to the first request, and what is in response to the second request (which the server may not be expecting and clearly doesn’t reply to).

If the server really is going to be that stupid, at least the packets it sends back should be delimited in some way. If the server responded , the client could tell when the first packet ended, and the second began.

Then, the client would not need to send two requests.

But, since you haven’t told us anything about the server, this is all supposition on my part.

Hi PaulS, I'll better explain what the server is supposed to do.

The server is not so stupid, the example I posted is just an example, just to show what kind of error happens to me.

Real communication is a protocol, where messages are exchanged. Every message has two parts: a header and a payload. So the two messages are not two dummy strings (like in the example I posted), they are two byte buffers instead.

Real communication happens this way:

  • the server listens for incoming connections (of course)
  • the client sends the first bunch of data ("First client message" in the previous post)
  • the client sends the second bunch of data ("Second client message" in the previous post)
  • the server parses the data (first and second client messages), controls their integrity, then prepares a reply, formatted as the client one (first + second message, here as well)
  • the server sends its response ("First/Second server message" in the previous post). It mimics the client logic, using a two steps communication.

I hope my explanation was clear and exhaustive!

Thanks in advance.

Real communication happens this way:

  • the server listens for incoming connections (of course)
  • the client sends the first bunch of data ("First client message" in the previous post)
  • the client sends the second bunch of data ("Second client message" in the previous post)
  • the server parses the data (first and second message), controls their integrity, then prepares a reply (first + second message, here as well)
  • the server sends its response ("First/Second server message" in the previous post).

I didn't expect that the text strings you were sending in the example represented any real payload.

This post confirms that communication with the server is asynchronous. It says that the client sends data twice, without waiting for a response.

The server sends a single response. However, it appears that the server sends a response before the client has sent the second request/message.

So, the fault still appears to be with the server sending a reply prematurely.

Perhaps the client should be modified to send all the data at once, instead of in two packets.

Or, the server should not respond until it has all the data it needs.

Since the server is (typically) stateless, I think the easiest change is to have the client not send anything until it has ALL the data that it needs to initiate communication.

Either that or I've missed something...

However, it appears that the server sends a response before the client has sent the second request/message.

This happens only in the example from my first post. The real server send its requests (header + payload) only after it received the client requests.

So, the fault still appears to be with the server sending a reply prematurely.

See above.

Or, the server should not respond until it has all the data it needs.

See above.

Perhaps the client should be modified to send all the data at once, instead of in two packets.

This is theoretically possible, but can lead to mistaken behaviours, for two reasons:

  • Data are sended over the net. Using the header+payload method, you can get overhead under control. If the header is broken, the receiving socket can close the communication (or request a new one) at once.
  • header+payload can (in theory) exceed the 2048 bytes limit. If I remember correctly, 2048 bytes is the default buffer limit for the W5100 chip, mounted on the eth shield. For a twist of fate, my maximum payload length can be...2048 bytes!

Since the server is (typically) stateless, I think the easiest change is to have the client not send anything until it has ALL the data that it needs to initiate communication.

This really happens, but data are splitted for the above reasons.

This happens only in the example from my first post. The real server send its requests (header + payload) only after it received the client requests.

After receiving BOTH requests? From the example posted, this is not the case.

header+payload can (in theory) exceed the 2048 bytes limit.

Perhaps it's time to stop dancing around and get down to specifics. What is the client sending that can exceed a long paragraph in length?

This post, for instance contains 602 bytes. Your request to the server (each of them) would need to contain nearly 4 times as much information.

After receiving BOTH requests? From the example posted, this is not the case.

I’m sorry PaulS, but the code from my first post does not perfectly match with the ‘real’ server/client communication, that I reassume here:

  • the server listens for incoming connections (of course)
  • the client sends the first bunch of data (“First client message” in the previous post)
  • the client sends the second bunch of data (“Second client message” in the previous post)
  • the server parses the data (first and second client messages), controls their integrity, then prepares a reply, formatted as the client one (first + second message, here as well)
  • the server sends its response (“First/Second server message” in the previous post). It mimics the client logic, using a two steps communication.

This example better fit with the above protocol:

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 33, 17 };
byte gateway[] = { 192, 168, 33, 13 };
byte server[] = { 192, 168, 33, 16 };  // notebook

Client client(server, 8001);

void setup() {
  Ethernet.begin(mac, ip, gateway);
  Serial.begin(9600);
  
  // give the Ethernet shield a second to initialize
  delay(1000);
  Serial.println("connecting...");

  if (client.connect()) {
    Serial.println("connected");
    Serial.println("- Sending first message -");
    sendClientMessage("First client message");
    Serial.println("- Sending second message -");
    sendClientMessage("Second client message");
    
    Serial.println("- Reading first message -");
    readServerMessage();
    Serial.println();
//    Serial.println("delay...");
//    delay(500);
    Serial.println("- Reading second message -");
    readServerMessage();
  } 
  else {
    Serial.println("connection failed");
  }
}

void loop()
{  
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();

    for(;;)
      ;
  }
}

void sendClientMessage(char *msg) {
  client.println(msg);
}

void readServerMessage() {  
  while (client.available()) {
    char c = client.read();
    Serial.print(c);
  }  
}

Your request to the server (each of them) would need to contain nearly 4 times as much information.

I only said that my maximum payload length is 2048 bytes, and that the header+payload length therefore CAN theoretically exceed the 2K limit. A message of mine has not an average length: it can vary from a little bytes to extremely long payload. It only depends from how much data it contains.

With the code posted above, this is the output from my serial monitor:

connecting...

connected
- Sending first message -
- Sending second message -
- Reading first message -

First server messageSecond server message
- Reading second message -

disconnecting.

Uncommenting the 'delay(500)' line, the output is the following:

connecting...

connected
- Sending first message -
- Sending second message -
- Reading first message -
First server message
delay...
- Reading second message -

disconnecting.

I can't explain why the delay makes any difference.

I also don't understand why (or whether) the server is actually sending two different packets.

Finally, I don't understand why the server is not using start-of-packet and end-of-packet markers to delimit each response, regardless of whether there is actually one or two packets.

I'm not picking on you, I hope you understand, and I have enjoyed the dialog. I just don't understand some things about what you are doing. I can only presume that you do not want to share the server code for your own reasons.

Why would the server think the below would be sending two seperate messages? I would think it would be seen as a single string of data. The same for the data received back, it is just a string of data that needs to be parced and interperted. Looks like the only thing that is a data delimiter in the communication is the connection open/close.

Serial.println("connected"); Serial.println("- Sending first message -"); sendClientMessage("First client message"); Serial.println("- Sending second message -"); sendClientMessage("Second client message");

Looks like the only thing that is a data delimiter in the communication is the connection open/close

It also appears that the connection is closed even if there is pending data to be read. Once the connection is closed, the pending data is flushed, if I remember correctly.

I also don’t understand why (or whether) the server is actually sending two different packets.

What the server does:

  • receives the message header, which has fixed length
  • parse the header to ensure correctness and to know how many bytes it must read (payload length is one of the header’s fields)
  • receives the message payload
  • does stuff
  • prepares a response message
  • sends the response message’s header
  • sends the response message’s payload

Finally, I don’t understand why the server is not using start-of-packet and end-of-packet markers to delimit each response, regardless of whether there is actually one or two packets.

I don’t understand what ‘start-of-packet’ and ‘end-of-packet’ mean. Can you explain me, please?

I can only presume that you do not want to share the server code for your own reasons.

Here I post the example server code, that is a standard C socket. The only difference with the real server code is that the latter parse the client message, then send an appropriate response message (see the first answer above).

/* A simple server in the internet domain using TCP
   The port number is passed as an argument */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h> 
#include <sys/socket.h>
#include <netinet/in.h>

void error(char *msg)
{
    perror(msg);
    exit(1);
}

int main(int argc, char *argv[])
{
     int sockfd, newsockfd, portno, clilen;
     char buffer[256];
     struct sockaddr_in serv_addr, cli_addr;
     int n;

     if (argc < 2) {
         fprintf(stderr,"ERROR, no port provided\n");
         exit(1);
     }
     sockfd = socket(AF_INET, SOCK_STREAM, 0);

     if (sockfd < 0) {
        error("ERROR opening socket");
     }
     memset((char *) &serv_addr, 0, sizeof(serv_addr));
     portno = atoi(argv[1]);
     serv_addr.sin_family = AF_INET;
     serv_addr.sin_addr.s_addr = INADDR_ANY;
     serv_addr.sin_port = htons(portno);

     if (bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0) {
        error("ERROR on binding");
     }
     listen(sockfd, 5);
     clilen = sizeof(cli_addr);
     newsockfd = accept(sockfd, 
                 (struct sockaddr *) &cli_addr, 
                 (socklen_t *) &clilen);

     if (newsockfd < 0) {
        error("ERROR on accept");
     }
     memset(buffer, 0, 256);
     n = read(newsockfd, buffer, 255);

     if (n < 0) {
        error("ERROR reading from socket");
     }
     printf("Here is the message: %s\n", buffer);

     memset(buffer, 0, 256);
     n = read(newsockfd, buffer, 255);
     if (n < 0) {
        error("ERROR reading from socket");
     }
     printf("Here is the message: %s\n", buffer);

     memset(buffer, 0, 256);
     strncpy(buffer, "First server message", 21);
     n = write(newsockfd, buffer, strlen(buffer));
     if (n < 0) {
        error("ERROR writing to socket");
     }

     //sleep(1);
     memset(buffer, 0, 256);
     strncpy(buffer, "Second server message", 22);
     n = write(newsockfd, buffer, strlen(buffer));
     if (n < 0) {
        error("ERROR writing to socket");
     }

     return 0; 
}

You’ll notice that in the messages we send back and forth that there are delimiters all over the place. There are spaces between words, and punctuation marks between sentences, and white space between paragraphs.

Each sentence starts with a capital letter. These conventions make it easier to read text, thanwhenitispresentedlikethis.

These delimiters are start of packet and end of packet markers. The server should be sending something like . The start of packet marker in these examples is <, and the end of packet marker is >.

With start and end of packet markers, the client code can tell where the first response starts and ends, and when the second response starts and ends.

In the last reply, you imply that the server is sending one response per client packet. That is a change from previous posts, where you say that the server is sending multiple replies to one or more client requests.

So, for clarification, how does the server know when the client has sent enough data?

Does the server respond to each client request, or does it only respond to the last request, when the request is complete?

Does it respond with one reply, or with more than one reply? Does the reply contain enough information so that the client can tell when has received the complete response?

In the last reply, you imply that the server is sending one response per client packet. That is a change from previous posts, where you say that the server is sending multiple replies to one or more client requests.

When I wrote:

sends the response message (header + payload)

I meant that the server uses the same logic as the client: it first sends header, then sends payload. Maybe I was not so clear, sorry.

So, for clarification, how does the server know when the client has sent enough data?

  • the server uses the first read() to read the message header. Now, using header->len, it knows how much payload it must receive.
  • the server uses the second read() to read the message payload.

Same logic applies at the other end (from server to client).

Does the server respond to each client request, or does it only respond to the last request, when the request is complete?

The server starts to send the response message only after it has received the entire message from the client. So, as you say, it replies only when the request is complete.

Does it respond with one reply, or with more than one reply? Does the reply contain enough information so that the client can tell when has received the complete response?

The server's response message is formatted exactly like the client message: a header plus a payload. The logic to know when to stop reading is the same as explained above.

So, the client should read the reply in a single operation. Why are there, then, two calls to read the reply?

I can rewrite my initial question like this:

Is there a way, using the Arduino Ethernet library, and using the Arduino like a client, to receive a message made up of a header and a payload? The server sends them separately, but using the same connection.

How can this be implemented?

Thanks in advance.

The server sends them separately, but using the same connection.

So what constitutes the "separately" in the transmission? What is the separator?

So what constitutes the "separately" in the transmission? What is the separator?

Zoomkat, you're right. There's no separator character in this transmission: simply, each end 'knows' how much bytes to read (the header fixed length + the payload length, founded into the field header->len).

Zoomkat, you're right. There's no separator character in this transmission: simply, each end 'knows' how much bytes to read (the header fixed length + the payload length, founded into the field header->len).

Then the answer is easy. Collect the incomming data into a string and extract the desired data from the string.