Client.read() loop is emptying the buffer faster than character can arrive

Hello,

As the subject implies, I have had some trouble communicating with my Arduino + Ethernet shield.

I have configured my Arduino to connect as a server in the below sketch, and I am sending it a string of characters over Telnet (using PuTTY). The sketch then saves that string and simply prints it straight back to the client.

/*

A simple Telnet server that receives text input
via Telnet and prints it back to the client.

Useful for testing string capture and comparison
methods.
 
 */

#include <SPI.h>
#include <Ethernet.h>

#define  bufferSize  10

// declare global parameters
char incomingMessage[bufferSize];
EthernetClient client;

// Eddie's Ethernet shield MAC address
byte mac[] = { 0x90, 0xA2, 0xDA, 0x0E, 0xA7, 0xAA };

/*
// Static IP addresses for connecting to Eddie's home ethernet network
IPAddress ip(192,168,254,200);
IPAddress dnServer(192,168,254,254);
IPAddress gateway(192,168,254,254);
IPAddress subnet(255,255,255,0);
*/

// Initialize the Ethernet server library
// with the IP address and port you want to use 
// (port 80 is default for HTTP):
EthernetServer server = EthernetServer(80);

void setup() {
 // Open serial communications and wait for port to open:
  Serial.begin(9600);

  // start the Ethernet connection and the server:
  Ethernet.begin(mac);
  server.begin();
  Serial.print(F("Arduino is connected on IP address: "));
  Serial.println(Ethernet.localIP());

}

void loop() {
  // listen for incoming clients
  client = server.available();
  // if a client is present, collect string command sent
  // and print to client
  if (client) {
    readDataFromClient();
    // print out string command
    client.println(incomingMessage);
  }    
}

// Checks for user input, and saves to incomingMessage[] global array
void readDataFromClient() {
    // initialise array index counter
    byte i = 0;
    // record message
    while (client.available() > 0 && i < bufferSize) {
      // the below delay needed to ensure correct behaviour
      //delay(10);      
      // read and store new byte
      incomingMessage[i] = client.read();
      // check if end of incoming message
      if (incomingMessage[i] == '\r') {
        // replace carriage return with null terminator
        incomingMessage[i] = '\0';
        // set array index counter to end
        i = bufferSize;
        // flush out any remaining control characters which would
        // stay in client buffer until next loop
        client.flush();
      }
      // increase array index counter
      i++;
    }
}

The sketch above works fine, but only because after a long debug session I realised that on some occasions the "record message" loop that starts on line 65 is cut short prematurely. I imagined it must be because the client.read() loop is going faster than the rate of arrival of the characters from my Telnet client, so I inserted a small delay in line 67 (commented out in the sketch above, just put it back in to stop seeing strange behaviour)

The funny thing is, this "strange behaviour" exhibited by the loop being exited prematurely only appears if I try to send commands of 1-3 characters in length. If I send a command of 4-9 characters in length then everything behaves fine, even without the delay.

And so I feel that -even though I appear to have solved the problem by inserting that small delay in line 67 - I'm not sure I actually understand the problem correctly, and so I can't know if this fix is going to work in all cases. Is there someone out there that knows exactly what is causing the behaviour described here, and can help me understand the best practice way to fix this?

Many thanks in advance for any help offered.

Ed

Maybe the command is being sent in more than one packet? Maybe you should wait for a terminator character. Are you sending a CR/LF with each command? If so, wait for the CR instead of using client.available().

If so, wait for the CR instead of using client.available().

I don't think that "instead of client.available()" is correct. OP is right. The Arduino can read data faster than it arrives. But, it makes no sense to read if there is nothing (yet) to read.

That's why client.connected() and client.available() are (normally) used together. As long as the client is connected, the server has not sent all the data. So, you need to wait for the rest of the data.

Read everything that is available, but do not give up on the server until is says it's done sending data.

PaulS:
I don't think that "instead of client.available()" is correct. OP is right. The Arduino can read data faster than it arrives. But, it makes no sense to read if there is nothing (yet) to read.

That's why client.connected() and client.available() are (normally) used together. As long as the client is connected, the server has not sent all the data. So, you need to wait for the rest of the data.

Read everything that is available, but do not give up on the server until is says it's done sending data.

The first is not true. The ethernet shield can receive data a 10MHz. The SPI line is good only to 8MHz, and unless you change it, it is set to 4 MHz by default.

I will grant you the server should close the connection tho. My bad.

edit: I did forget one thing tho. If you are testing server code, use Raw mode on PuTTY, not telnet.

The first is not true. The ethernet shield can receive data a 10MHz. The SPI line is good only to 8MHz, and unless you change it, it is set to 4 MHz by default.

What the shield can do, and what the ISP/network/etc. that it is connected to can do are two (very) different things.

The Arduino can read data faster, under some circumstances, faster than the ISP/server/network can deliver it.

So, the Arduino SHOULD check that there is data to be read before attempting to read data.

PaulS:
What the shield can do, and what the ISP/network/etc. that it is connected to can do are two (very) different things.

The Arduino can read data faster, under some circumstances, faster than the ISP/server/network can deliver it.

So, the Arduino SHOULD check that there is data to be read before attempting to read data.

It receives data packets faster than the SPI line can transfer them to the Arduino in most cases, but not all. If the packets are small, the packet header can be larger than the payload.

However, if the data is coming in multiple packets, you must empty the w5100 rx buffer before the w5100 will send an ACK for that packet, and then the sender will send another packet. That is when there will temporarily be no characters in the rx buffer. Between packets.

Yes, you should check that there are characters in the rx buffer before reading them.

Thanks all for your interesting discussion points.

In theory then, if I change the loop from:

   while ( client.available() && i < bufferSize) {
      // rest of loop code
    }

to...

   while ( client.connected() && i < bufferSize) {
      // wait to make sure data has arrived
      if( !client.available() ) 
        client.println('X');  // Interesting to see how many times this
                              // loop goes through before next character arrives

      // rest of loop code
    }

...then this should fix the problem for any lag of unknown length between the client connecting and the characters being sent. If I run the new code, I can also see the occasional 'X' being printed to show that the delay loop is indeed being put to use. However, in this new version I now occasionally get a couple of funny characters being returned over the client. They look like this: ▒ and normally appear at the end of the string being returned. When they appear, a list of around 8 'X's being printed also show that it was necessary to run the delay loop quite a few times during for that particular command, whereas normally if I get an 'X', it's just the one. These strange characters only appear when using the new code above, and do not appear if I used a simple delay to wait for the characters (as suggested in my original sketch post).
Any idea why?

My use of client.connected() as a condition of the while loop above is arguably unnecessary I suppose, as the rest of the loop code (not shown above for clarity, but shown in the original post) looks for a carriage return to appear, before replacing it with a null terminator, getting out of the loop and running client.flush(). However, as I'm not sure if I understand what client.connected() actually does, I left it in for robustness sake. Can anyone recommend reading an in-depth, easy-to-understand, explanation about what the Ethernet library actually does? I would like to learn more! After which I'd be happy to make some suggestions as to how the documentation of this library could be expanded in the Arduino reference to help beginners understand how best to use it.

Your initial post threw me off a bit.

/*

A simple Telnet server that receives text input
via Telnet and prints it back to the client.

Useful for testing string capture and comparison
methods.

*/

Here is my web server code. You might want to look at my old server code on this page.
http://playground.arduino.cc/Code/WebServerST
It has most of the stuff you need, including a timeout feature. It does not include the socket freeze fix that the new server code has tho.

You are on the right track with your new code.

edit: You should wait for a newline character to insure you get at least the first line of the request header. zoomkat's server code uses that approach. My server code waits for a blank line to determine "end of header". That way I can also process POST requests if I need to. It also insures the w5100 rx buffer is empty so the socket will close correctly.

Thanks SurferTim. I used the word "Telnet" in my original sketch because the Ethernet sketch examples included with the Arduino UI are what I originally based a lot of my code on, and so I thought that was the correct word to use. Sorry about that.

For the record, the "improved" loop I uploaded in my last post is no good because I hadn't realised client.connected() is true even after client.flush() has been run. So the sketch quickly gets stuck in the delay loop, waiting for the next characters to arrive, and preventing you from running any other processes.

The code you point to looks like a great repository of example code, so I'll get stuck in to that and use it to learn about best practices when using this library. Thanks a lot for that!

It is the server's responsibility to close the connection when it is finished sending packets. If you look carefully at my code, you will find I check client.connected in my server code to insure the client hasn't closed the connection, but it is the client.stop() call in the server code that closes the connection. That will cause the client.connected() loop to exit.

I had a look through your WebServer sketch and have already taken away a few useful things. Thanks for that.

I've looked further into this problem of client.available() occasionally returning zero when there is still supposed to be data coming in. Inserting some delays definitely solves the problem. In your WebServer sketch is it possible the problem doesn't emerge simply because there are quite a few instructions slowing the loop between each call to client.read()? Or am I missing an active precaution that you took to avoid this problem?

With regards to your client.stop() suggestion, as I'm actually hoping to achieve a sort of "command line" interface through PuTTY, I can't use client.stop() because that disconnects you from your PuTTY connection. Sorry if the "command line" intention wasn't clear from the start.

If you are using PuTTY it should send a separate packet every time you press the enter key. My code takes a blank line. That is two enter keys in a row.

Ok, I will see how I can implement this in my sketch. Thanks again for your help SurferTim!

With regards to your client.stop() suggestion, as I'm actually hoping to achieve a sort of "command line" interface through PuTTY, I can't use client.stop() because that disconnects you from your PuTTY connection. Sorry if the "command line" intention wasn't clear from the start.

I missed this part yesterday. It was until you changed it. Telnet is a command line interface. It will stay connected during a session until you enter a command like "QUIT" or "EXIT". That triggers the client.stop().

That was the point of waiting for a newline character. That is the enter key. That signals the end of the command, then you process that line.