@fungus I think you need to be realistic about what can be done with an 8 bit processor with 2k of ram running at 16mhz.. i would say 30kb/s it's very impressive for such a simple device. if you want more power you might need to get a Linux single board computer.
I've done some more tests and I found that the fastest I could transmit data across a network connection from the Arduino is about 75kbyes/sec. I think my web server can be optimized a little bit more... :-)
To me it seemed that 200 clock cycles per byte sent is quite a lot so I looked in w5100.cpp and the loop to send data looks like this:
for (int i=0; i<_len; i++)
SPI.transfer(_addr >> 8);
SPI.transfer(_addr & 0xFF);
So it does a complete SPI transaction per byte ... there's the problem!
Can the w5100 not receive bytes sequentially?
I don't really need this to be a serious web server, I'm just experimenting. The main idea behind what I'm doing is to create a web server with extensions to view/control the Arduino pins directly. That way I can do things with the Arduino via a web browser (eg. my mobile phone) from anywhere in the world.
But ... it would be nice if it worked well as a little web server too, there's plenty of space on the SD card for files.