W5100 Network shield - overcome 2kB receive limit?

Projects Networking, Protocols, and Devices
1

Hi,

I'm using an Arduino Wiznet Ethernet W5100 shield with an UNO.

It's set up and running and configured to receive data from a web browser via the POST method and dump it 1 character at a time to the serial monitor.

It works but I've run into a ~2KB receive limit. Posting a data string larger than 2KB results in data just being lost.

Looking to send about 250KB (quarter of a MB) back from the browser to the UNO via the W5100.

I'm using the client.available() and client.read() methods to test if there is data to read, and then read it. I works for data streams less than 2KB.

There are other posts on this but no resolution:

http://forum.arduino.cc/index.php?topic=19260.0
http://forum.arduino.cc/index.php?topic=444493.0

I understand that there is only an 16KB buffer on the 5100 (source:datasheet), and that this is shared across 4 sockets (4KB per socket), but cannot see why this would limit received data on a TCP\IP connection.

TCP\IP is a stateful protocol, meaning there is a handshake between nodes (unlike UDP\IP which is just broacast). It allows for traffic negotiation (flow control), so the receive buffer size shouldn't be an absolute problem (not dropped packes when the buffer is full anyway). The data sheet for the 5100 states:

The W5100 includes fully hardwired, market-proven TCP/IP stack and integrated Ethernet
MAC & PHY. Hardwired TCP/IP stack supports TCP, UDP, IPv4, ICMP, ARP, IGMP and PPPoE

So the TCP\IP stack is implemented in hardware, and the designers must have known about the receive buffer limit when the stack was implemented, so why are packets just dropped when the buffer is full? As 'westfw' states in the first post, ' a TCP window smaller than the available buffer memory (should be set), which should prevent the remote host from ever sending a packet that won't fit'.

TCP windows are set at the stack level right? Is there any way to set it?

If this really is a stack implemention 'feature' on this chip, are there any ways round it?

I tried playing with the memory allocation for sockets (the datasheet does elaborate on this a bit). Modified .cpp code and you do get a bit more recieve buffer, but not much and that's still at the expense of available sockets if allocation is high.

Another option would be to open and close multiple connections in series, posting <2KB each time. Not standard browser behaviour though.

Any suggestions appreciated.

Thanks,

2

I don't know what you are talking about. The packet buffers in the w5100 will receive one packet at a time, and will not send an ACK until you have read the entire contents of the rx buffer. Then the sender will send the next packet, and it will not get an ACK for that packet until, once again, you empty the rx buffer.

If you are limited to receiving 2K, then that is the fault of your code.

3

Hello SurferTim,

OK. I just tried sending data using putty in RAW mode on port 80 (instead of using a browser), and the code works without limitation, so it's not the tcp\ip stack on the W5100. I should have tried that first.

Something to do with browser behaviour then (most probably the HTML I'm feeding it...).

Thanks for your help.

4

Are you using client or server code on the Arduino? I have examples in the playground that might give you some ideas.

If it is client code
http://playground.arduino.cc/Code/WebClient

If it is server code
http://playground.arduino.cc/Code/WebServerST

5

Hi, I'm using server code (I believe);

It's simple HTML, Can't for the life of me understand why the data that's sent back is truncated. The truncations are browser dependent also - vary slightly between one browser and the next (IE, FireFox, QTWeb).

The code below compiles. If you're curious, just dump 20k of text into the form field and watch the console window (19200 baud).

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};
IPAddress ip(169, 254, 1, 1);
EthernetServer server(80);

void setup() {
  Serial.begin(19200);
  while (!Serial) {
    ;
  }
  Ethernet.begin(mac, ip);
  server.begin();
  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());
}


void loop()
{
  // listen for incoming clients
  EthernetClient client = server.available(); // evaluates to true when hardware ethernet connection established with client and at least 1 packet sent
  if (client) {
    Serial.println("Client connected "); Serial.println();
    boolean currentLineIsBlank = true;    // an http request ends with a blank line
    while (client.connected()) { // if the client is still connected via tcp-ip
      while (client.available()) { // while there are bytes available to read, as sent by the client
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline character) and the line is blank, the http request has ended,so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // POST data follows at this point
          while (client.available())
          {
            char c = client.read();
            Serial.write(c);
          }
          Serial.println();

          Serial.println("Sending browser response");
          // send a standard http response header
          client.println("HTTP/1.0 200 OK");
          client.println("Content-Type: text/html");
          client.println();
          //form added to send data from browser and view received data in serial monitor
          client.println("<HTML><BODY><B>Data Input</B>
");
          client.println("<FORM ACTION=\"http://169.254.1.1:80\" METHOD=\"POST\" enctype=\"text/plain\" >");
          client.println("Data: <INPUT TYPE=\"TEXT\" NAME=\"Email\" VALUE=\"\" SIZE=\"50\" MAXLENGTH=\"25000\">
");
          client.println("<INPUT TYPE=\"SUBMIT\" NAME=\"submit\" VALUE=\"Submit\">");
          client.println("</FORM> 
 </BODY> </HTML>");

          client.stop();
        }
        else if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        }
        else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      } // while(client.available())
    } //while (client.connected())
    Serial.println("Disconnected");
  }
}

The console data that's sent does seem to recognise the amount of data entered (Content-Length: 20503) but data that's read by client.read() is truncated.

Thanks,

6

It isn't truncated. It is being sent in more than one packet. You are reading only the first packet. If it is a POST request, you must parse the request for the Content-Length, and read that many characters after the blank line.

7

Hello SurferTim,

Yes you are corect. I've changed the code to read back the reported number of characters and it works.

The posted data seems to come in bursts - there's a small delay between chunks of form data but that's not a problem.

Wouldn't have figured this out without your help.

Thanks.