Ethernet Shield sending byte by byte?

All right, I think I have it figured out. If you try to connect again before the connection has completely ended, you will start another connection. This is fine as long as you don't go above four connections. The easiest way to not deal with connection pool management is to just wait until your current connection is really closed.

The act of checking the status register actually slows down the disconnect cycle. The code below should be placed after you have finished receiving all your data:

client.stop();
while(client.status() != 0) {
   delay(5);
}

Note that I'm using the status() function which is not exposed on the Ethernet library reference. This is a direct access to the W5100 Sn_SR status registers; all you need to know is that !client.connected() may not actually signify an ended connection. The delay(5) is necessary to give the W5100 some breathing room while it tries to terminate the connection.

Using this method, I am able to reliably download a web page about twice per second, probably a limit of the SPI and my processing routines. The time between downloads varies somewhat, but does not go over 1 second.

For ultimate reliability, you may want a timeout in the above code in case it somehow gets stuck waiting for a connection that never closes.