Ethernet Shield sending byte by byte?

Yes, I have tryed Ethernet2 and I got the same result, one byte sent from the server and then a response (Ack) from the client and then next byte ..... and so on.

A quick look at this suggests the reason for the issue is that the client send code is implemented on top of the Print class and at the bottom of the "stack" is a routine that prints a byte. And to print a string that routine gets called multiple times.

I don't know if it's possible to override the higher-level methods but it would seem worthwhile as this must be adding a fair bit of overhead. The underlying ethernet code is certainly capable of sending multiple bytes.

--Phil.

I took a look at this since I was using WireShark to track down another issue (with my code/Adobe Flash not the library :-)).

I can confirm what the OP posted, in my case for each byte of data sent via println 60 bytes were actually being sent on the wire--this includes for the response headers etc.

Unfortunately trying to solve this raises the whole issue of buffering etc which means it's not quite a straight fix of println. Although if the data isn't in PROGMEM then it's probably more straight-forward but that's hardly ideal.

Will ponder.

--Phil.

Reason for me to care about this is the communication cost. As you have seen there are a lot of overhead. I have the Arduino stuff in my boat and connected to a mobile network and I have to pay extra if I send to much data. Unfortunately I am not a programmer so I can't contribute with coding skills. I hope you can find a solution

Thomas in Stockholm

Hi,

I've taken a look at the code and have produced a modified copy of the Client library--as found in Arduino 0013--the new code provides better Arduino Ethernet transmission efficiency.

It's really only an interim solution and will only work if you make use of the println or print methods and supply a string.

I've only tested it with the standard WebServer example but that resulted in ~75% fewer bytes being transmitted. Even in this scenario there's still a lot of overhead as the numerals are still transmitted one byte at a time. Much more detail and analysis at the link above.

Please let me know how this works for you.

--Phil.

I can't get your code ?? Is your site down?

Thomas

I can't get your code ?? Is your site down?

Unfortunately, it appears so. That was bad timing. :-/

Put this at the end of Client.cpp:

#if 1
#include <string.h>
void Client::print(const char c[]) {
  send(_sock, (uint8_t *) c, strlen(c));  
}

void Client::println(const char c[]) {
  print(c);
  print("\r\n");
}
#endif

And then this in the appropriate place (use the first and last lines to show you where) in Client.h:

  friend class Server;
#if 1
  void print(const char c[]);
  void println(const char c[]);
  using Print::print;
  using Print::println;
#endif
};

--Phil.

Very interesting.. number of sent/received bytes i exactly the same 444 in and 589 out from the Arduino. But number of interactions went down from 888 to 136, very good!!!

Thomas ;D

Great, thanks for the feedback.

--Phil.

Some more observations. As every PRINT and PRINTLN generate a "segment" "on the line" I have tryed to reduce the numbers of these statement and instead put in as many chars as possible per PRINT, about 10 to 20. But there seems to be some limitation in the code or the Arduino HW because I got some hang up in the communication at some point. Very strange, I reduce the number of PRINT but more data per PRINT up to some limit ???

Thomas :-/

One more observation, when sending from a buffer like:
Char buf1[300]; and client.print(buf1); there seems to be a limit at 240-250 chars. I still use your new code Phil.

Thomas

Additional information: apparently no one involved in developing the Ethernet Library has tested communications with web servers other than Apache. The library as-is will not communicate with lighttpd. Phil's modification above does allow communication, you will need to take care to render each line into a complete string before trying to println() it; multiple print() commands per line will not work.

So now I need to figure out why the Ethernet library hangs after four connection attempts at 5 second spacing, even with stop() command after each connection. At this point I know the connection idles in the FIN_WAIT state for less than a second up to 20 seconds. If another connection is attempted during this time, apparently the library makes a new one...it's limited to four. The Client::connected() function does not address this at all.

All right, I think I have it figured out. If you try to connect again before the connection has completely ended, you will start another connection. This is fine as long as you don't go above four connections. The easiest way to not deal with connection pool management is to just wait until your current connection is really closed.

The act of checking the status register actually slows down the disconnect cycle. The code below should be placed after you have finished receiving all your data:

client.stop();
while(client.status() != 0) {
   delay(5);
}

Note that I'm using the status() function which is not exposed on the Ethernet library reference. This is a direct access to the W5100 Sn_SR status registers; all you need to know is that !client.connected() may not actually signify an ended connection. The delay(5) is necessary to give the W5100 some breathing room while it tries to terminate the connection.

Using this method, I am able to reliably download a web page about twice per second, probably a limit of the SPI and my processing routines. The time between downloads varies somewhat, but does not go over 1 second.

For ultimate reliability, you may want a timeout in the above code in case it somehow gets stuck waiting for a connection that never closes.

Additional information: apparently no one involved in developing the Ethernet Library has tested communications with web servers other than Apache. The library as-is will not communicate with lighttpd. Phil's modification above does allow communication, you will need to take care to render each line into a complete string before trying to println() it; multiple print() commands per line will not work.

Hmmm, that's interesting. Is that standards compliant behaviour? I would have thought it didn't matter how the bytes arrived to the server, just that they did...

--Phil.

I don't pretend to know the standards...just that Apache tends to be way more forgiving than some other servers. I ran into another problem with lighttpd on an unrelated (XPort) issue. In that case, the AF_XPort library was definitely doing something wrong, but Apache would ignore it.

I have gone on to test lighttpd a little more, and while it definitely didn't work at one byte per packet, it seems to work with at least a few bytes per packet. Have not determined the lower limit, but several chunks per line seems to work.

Since an IP network can fragment packets enroute (e.g. if an intermediary network has a smaller MTU than the originator), it seems to me that any server that can't deal with fragmented packets is broken.

The arduino ethernet code does need some work in this area (and that seems to be taking place), but I don't think we can blame this one on the arduino.

-j

Well, lighttpd works fine with all the web browsers and runs huge installations, like Sourceforge, Youtube, Wikipedia, Myspace, Imageshack, etc. Arduino is the hack here, not lighttpd, so I'm inclined to blame us first.

I'll do more testing and try to narrow the exact cause of the problem, it may have even been my code at the time and maybe I fixed two things at once.

I'm having similar problems with Apache, I tried it on multiple servers, no luck..

I managed to use your client.status() to track down part of the problem, see this topic:

http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1238295170

If someone needs an ugly workaround, without touching the libraries - just use a proxy :slight_smile: