W5500 ethernet... how in the hell it sends 10GB per month?!

Hi, I have a "smart home" (I hate that term) project which needs internet connectivity.

It uses a common W5500 Ethernet controller and it works just fine. Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST. This is roughly 250MB of useful data per month.

Surprise Motherf*cker! It eats through 2GB data plan in a week. It needs roughly 10GB per month!

How is this possible? Where is the 9,75GB per month coming from? Is the HTTP POST really that data hungry with small frequent payload data?

Is there anything I can do to lower the data usage? * I can not make the data string smaller. * I need the 5s refresh rate. OK maybe not 5s, 10s would be also fine but no more. * I can not cache the data and send a big package every a minute or so - not enough memory. * It works with a standard Arduino Ethernet lib. * It does not sends the same data 100x, nothing like that. I do not touch the W5500 between the 5s period.

How does the Arduino Ethernet library interacts with the W5500? Does it support multi-byte SPI transfers or does it send each byte separately?

How do you think those "useful" data bytes are send?

They are sent through a protocol stack. Each layer of the stack adds some bytes to the data and passes the data on to the next layer. These bytes describe the data what it is e.g. a UDP or TCP header, where the data comes from e.g. IP address and where the data is supposed to get to another IP address and other stuff.

Have a look at the following wiki pages if you are interested.

https://en.wikipedia.org/wiki/Internet_protocol_suite

https://en.wikipedia.org/wiki/Transmission_Control_Protocol

Guess what your internet provider charges you for all the bits you are sending and receiving. Yes, even if you just send data, data is sent back to you to let you know the data has arrived (not all protocol do that, but TCP does).

If you like to look at network traffic. You can install Wireshark on your PC and see the traffic from and to your PC. You can look at all the packages e.g. when you open a browser and load Google.

and every packat contains both mac addresses and small packets are padded to minimal payload length of 64 bytes. every TCP packet is acknowledged. creating and closing a connection with TCP requires request a response packet ...

show your client code. It can make a big difference in how many bytes will be transferred depending on your coding! Use as less client.print as possible, if you have a lot of values to transmit, consider to fill a buffer and transmit the buffer (or let's say in 5 chunks of 100 values each as SRAM is limited also on an UNO).

Hi,
this is the sanitized client code, it repeats every 5s, measurements happen in the meantime:

    client.stop();
    ServerStatus = client.connect(server_address, 80);
    client.print(F("POST /"));
    client.print(url_address);
    client.println(F(" HTTP/1.1"));
    client.print(F("Host: "));
    client.println(server_address);
    client.println(F("Content-Type: application/json"));
    client.println(F("User-Agent: arduino-ethernet"));
    client.print(F("Content-Length: "));
    client.println(calculated_msg_length);
    client.println(F("Connection: close\r\n"));
    client.print(MSG1_approx_50chars);
    client.print(MSG2_approx_300chars);
    client.println(MSG3_approx_50chars);
    client.println(F("Connection: close"));
    client.println();
    client.stop();

I guess I can merge few prints together. Will it make any difference?
Next week I try the Wireshark.

check with Wireshark in how many packets it is sent, but I expect one or two. you can use BufferedPrint from my StreamLib to buffer the prints. at least the SPI transfer to W5100 will be faster

checking with wireshark is a good idea, nevertheless I assume that each client.print produces a separate packet.
quick check: compare it with this
however, you could create a buffer for your HTTP header fields also.

;-(

noiasca:
checking with wireshark is a good idea, nevertheless I assume that each client.print produces a separate packet.

no it doesn’t

you are right, and I have no idea, where this assumption came from ...

edit:
Wireshark… interesting.

There is something rotten with the “F()” macro!

Everything sent with “F()” (for example client.print(F(“POST /”)); ) is sent char by char.
Everything sent without (for example client.print(url_address); ) is sent as a whole.

Will test and report.

bumerang123: edit: Wireshark... interesting.

There is something rotten with the "F()" macro!

Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char. Everything sent without (for example client.print(url_address); ) is sent as a whole.

Will test and report.

I recommend my BufferedPrint again

ok, that's all quite a mistery for me.

I tried this test sketch in 4 versions:

/*
  Web Server
*/

#include 
#include 

byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED
};
IPAddress ip(172, 18, 67, 96);

EthernetServer server(80);

#define VERSION 0          // 0 no, 1 reduced client print 
// 0 12912/725  --> 122 
// 1 12858/731  -->  71 
// 2 12926/611  --> 387 
// 3 13020/735  -->  19 

void setup() {
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  Serial.println("Ethernet WebServer Example");

  // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip);

  // Check for Ethernet hardware present
  if (Ethernet.hardwareStatus() == EthernetNoHardware) {
    Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
    while (true) {
      delay(1); // do nothing, no point running without Ethernet hardware
    }
  }
  if (Ethernet.linkStatus() == LinkOFF) {
    Serial.println("Ethernet cable is not connected.");
  }

  // start the server
  server.begin();
  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());
}


void loop() {
  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header

#if VERSION == 0
          client.println("HTTP/1.0 200 OK");          // changed
          client.println("Content-Type: text/html");
          client.println("Connection: close");        // the connection will be closed after completion of the response
          client.println();
          client.println("");
          client.println("");
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            client.print("analog input ");
            client.print(analogChannel);
            client.print(" is ");
            client.print(random(100, 1000));
            client.println("
");
          }
          client.println("");

#elif VERSION == 1
          client.print  ("HTTP/1.0 200 OK\r\n"          // changed  271/208
                         "Content-Type: text/html\r\n"
                         "Connection: close\r\n"        // the connection will be closed after completion of the response
                         "\r\n"
                         "\r\n"
                         "\r\n");
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            client.print("analog input ");
            client.print(analogChannel);
            client.print(" is ");
            client.print(random(100, 1000));
            client.println("
");
          }
          client.println("");

#elif VERSION == 2
          client.print  (F("HTTP/1.0 200 OK\r\n"          // changed
                           "Content-Type: text/html\r\n"
                           "Connection: close\r\n"        // the connection will be closed after completion of the response
                           "\r\n"
                           "\r\n"
                           "\r\n"));
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            client.print(F("analog input "));
            client.print(analogChannel);
            client.print(F(" is "));
            client.print(random(100, 1000));
            client.println(F("
"));
          }
          client.println(F(""));

#elif VERSION == 3
          char buffer[400];
          memcpy(buffer, '\0', 400);
          strcpy(buffer, "HTTP/1.0 200 OK\r\n"          // changed
                 "Content-Type: text/html\r\n"
                 "Connection: close\r\n"        // the connection will be closed after completion of the response
                 "\r\n"
                 "\r\n"
                 "\r\n");
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            strcat(buffer, "analog input ");
            char c[8];
            itoa(analogChannel, c, 10);
            strcat(buffer, c);
            strcat(buffer, " is ");
            itoa(random(100, 1000), c, 10);
            strcat(buffer, c);
            strcat(buffer, "
\r\n");
          }
          strcat(buffer, "\r\n");
          client.print(buffer);


#endif

          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
    Serial.println("client disconnected");
  }
}

Version 0 - Single client prints from RAM --> 122 |500x281

Version 1 - Less single client prints from RAM --> 71 |500x281

Version 2 - Less single client prints with F-Makro --> 387 |500x281

Version 3 - buffer - one client print --> 19 |500x281

interesting so far.

// V Flash/Globals                                          Incomming
// 0 12912/725  Single client prints from RAM               54 packets
// 1 12858/731  Less single client prints from RAM          43 packets
// 2 12926/611  Less single client prints with F-Makro     257 packets
// 3 13020/735  buffer - one client print                    6 packets

Have to adopt my wireshark settings. But at least I know I have to re-learn a lot...

version0.png|1920x1080

version1.png|1920x1080

version2.png|1920x1080

version3.png|1920x1080

bumerang123: Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char. Everything sent without (for example client.print(url_address); ) is sent as a whole.

You can see why here: https://github.com/arduino/ArduinoCore-avr/blob/1.8.3/cores/arduino/Print.cpp#L48-L53

size_t Print::print(const __FlashStringHelper *ifsh)
{
  PGM_P p = reinterpret_cast(ifsh);
  size_t n = 0;
  while (1) {
    unsigned char c = pgm_read_byte(p++);
    if (c == 0) break;
    if (write(c)) n++;
    else break;
  }
  return n;
}

and here: https://github.com/arduino/ArduinoCore-avr/blob/1.8.3/cores/arduino/Print.cpp#L62-L64

size_t Print::print(const char str[])
{
  return write(str);
}

so in the end: - if you want spare packets - reduce client.print - if you want to spare RAM - use F-Makro - but 'cause of the print implementation of the F-Makro don't use it with client.print - instead use strcpy_P / PSTR or Juraj's StreamLib

correct?

edit:

/*
  Web Server - Comparison of different client strategies
  https://forum.arduino.cc/index.php?topic=710547
  by noiasca
*/

#include 
#include 

byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};
IPAddress ip(172, 18, 67, 96);
EthernetServer server(80);

#define VERSION 5          // 0 no, 1 reduced client print 
// V Flash/Globals                                          Incomming
// 0 12912/725  Single client prints from RAM               54 packets
// 1 12858/731  Less single client prints from RAM          43 packets
// 2 12926/611  Less single client prints with F-Makro     257 packets
// 3 13020/735  buffer - one client print                    5 packets
// 4 13086/613  buffer with strcpy_P                         5 packets
// 5 13278/623  Streamlib chunks of 64 byte                 10 packets

#if VERSION == 5
const size_t RESPONSE_BUFFER_SIZE = 64;
#include                   // download in lib manager (by Juraj Andrassi)
#endif

void setup() {
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  Serial.println("Ethernet WebServer Example");

  // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip);

  // Check for Ethernet hardware present
  if (Ethernet.hardwareStatus() == EthernetNoHardware) {
    Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
    while (true) {
      delay(1); // do nothing, no point running without Ethernet hardware
    }
  }
  if (Ethernet.linkStatus() == LinkOFF) {
    Serial.println("Ethernet cable is not connected.");
  }

  // start the server
  server.begin();
  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());
}


void loop() {
  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header

#if VERSION == 0
          client.println("HTTP/1.0 200 OK");          // changed
          client.println("Content-Type: text/html");
          client.println("Connection: close");        // the connection will be closed after completion of the response
          client.println();
          client.println("");
          client.println("");
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            client.print("analog input ");
            client.print(analogChannel);
            client.print(" is ");
            client.print(random(100, 1000));
            client.println("
");
          }
          client.println("");

#elif VERSION == 1
          client.print  ("HTTP/1.0 200 OK\r\n"          // changed  271/208
                         "Content-Type: text/html\r\n"
                         "Connection: close\r\n"        // the connection will be closed after completion of the response
                         "\r\n"
                         "\r\n"
                         "\r\n");
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            client.print("analog input ");
            client.print(analogChannel);
            client.print(" is ");
            client.print(random(100, 1000));
            client.println("
");
          }
          client.println("");

#elif VERSION == 2
          client.print  (F("HTTP/1.0 200 OK\r\n"          // changed
                           "Content-Type: text/html\r\n"
                           "Connection: close\r\n"        // the connection will be closed after completion of the response
                           "\r\n"
                           "\r\n"
                           "\r\n"));
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            client.print(F("analog input "));
            client.print(analogChannel);
            client.print(F(" is "));
            client.print(random(100, 1000));
            client.println(F("
"));
          }
          client.println(F(""));

#elif VERSION == 3
          char buffer[400];
          memcpy(buffer, '\0', 400);
          strcpy(buffer, "HTTP/1.0 200 OK\r\n"          // changed
                 "Content-Type: text/html\r\n"
                 "Connection: close\r\n"        // the connection will be closed after completion of the response
                 "\r\n"
                 "\r\n"
                 "\r\n");
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            strcat(buffer, "analog input ");
            char c[8];
            itoa(analogChannel, c, 10);
            strcat(buffer, c);
            strcat(buffer, " is ");
            itoa(random(100, 1000), c, 10);
            strcat(buffer, c);
            strcat(buffer, "
\r\n");
          }
          strcat(buffer, "\r\n");
          client.print(buffer);
#elif VERSION == 4
          char buffer[400];
          memcpy(buffer, '\0', 400);
          strcpy_P(buffer, PSTR("HTTP/1.0 200 OK\r\n"          // changed
                                "Content-Type: text/html\r\n"
                                "Connection: close\r\n"        // the connection will be closed after completion of the response
                                "\r\n"
                                "\r\n"
                                "\r\n"));
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            strcat_P(buffer, PSTR("analog input "));
            char c[8];
            itoa(analogChannel, c, 10);
            strcat(buffer, c);
            strcat_P(buffer, PSTR(" is "));
            itoa(random(100, 1000), c, 10);
            strcat(buffer, c);
            strcat_P(buffer, PSTR("
\r\n"));
          }
          strcat_P(buffer, PSTR("\r\n"));
          Serial.print("|");
          Serial.print(buffer);
          Serial.print("|");
          client.print(buffer);

#elif VERSION == 5
          char buffer[RESPONSE_BUFFER_SIZE];
          BufferedPrint response(client, buffer, RESPONSE_BUFFER_SIZE);

          response.print  (F("HTTP/1.0 200 OK\r\n"          // changed
                             "Content-Type: text/html\r\n"
                             "Connection: close\r\n"        // the connection will be closed after completion of the response
                             "\r\n"
                             "\r\n"
                             "\r\n"));
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            response.print(F("analog input "));
            response.print(analogChannel);
            response.print(F(" is "));
            response.print(random(100, 1000));
            response.println(F("
"));
          }
          response.println(F(""));
          response.flush();

#endif

          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
    Serial.println("client disconnected");
  }
}
          char buff[128];
          BufferedPrint response(client, buff, sizeof(buff));
          response.println(F("HTTP/1.0 200 OK"));          // changed
          response.println(F("Content-Type: text/html"));
          response.println(F("Connection: close"));        // the connection will be closed after completion of the response
          response.println();
          response.println(F(""));
          response.println(F(""));
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            response.print(F("analog input "));
            response.print(analogChannel);
            response.print(F(" is "));
            response.print(random(100, 1000));
            response.println(F("
"));
          }
          response.println(F(""));
          response.flush();

|500x262

Sketch uses 15758 bytes (48%) of program storage space. Maximum is 32256 bytes. Global variables use 733 bytes (35%) of dynamic memory, leaving 1315 bytes for local variables. Maximum is 2048 bytes.

Screenshot from 2020-10-27 20-18-49.png|1360x715

Hi, quick update, sorry for not replying I don't have time to do anything.

client.print(F("Host: ")); - sends every character in separate packet client.print("Host: "); - sends one packet, BUT each client.print() sends separate packet client.println("Host: "); - same as previous BUT it sends additional packet for newline characters

I will try the buffered print from Juraj, thank you. At least for some of the prints - I don't have enough SRAM to buffer it all.

Original Arduino Ethernet lib is f*cking useless :( Will try another lib when I have time.