I'm writing a web server for my Ethernet Arduino to serve files from the SD card...it's looking pretty good.
It works fine when I access files from machines on the local network but if I try to access them across the Internet on my mobile phone the file sends hang after a few blocks of data have been sent.
The send loop is very simple, just read the file and send it in chunks of 64 bytes using a "Client" object.
I can "fix" the problem by putting in a delay between each block send, eg. "delay(50)". If I do that it all works 100% perfectly (although very slowly).
I'm starting to suspect there's a problem in the Ethernet library when the Arduino sends data faster than the other end can receive it. Does anybody know anything about this? Has anybody seen anything like this before?
Sounds like you are overflowing the ethernet transmit buffer.
I just finished a function for the ethernet library to avoid this problem. It allows you to check the free space in the transmit buffer. I have been testing it somewhat like the client.available() function. I call it client.free(). It returns the size of the unused transmit buffer.
SurferTim:
Sounds like you are overflowing the ethernet transmit buffer.
I would expect the the library/w5100 to wait if the buffer is full. This is TCP...
SurferTim:
I just finished a function for the ethernet library to avoid this problem. It allows you to check the free space in the transmit buffer. I have been testing it somewhat like the client.available() function. I call it client.free(). It returns the size of the unused transmit buffer.
I was thinking of doing something similar today. The Ethernet library has no non-blocking options at the moment...it definitely needs some.
I would expect the the library/w5100 to wait if the buffer is full. This is TCP...
I would too, but I do not see anywhere the buffer is protected from transmit buffer overflow. The TCP part doesn't start until you can get the data into the transmit buffer. The W5100 chip takes care of the rest of the TCP stuff from there.
Add this to /arduino-0022/libraries/Ethernet/client.h. I put it below the declaration for available(). virtual int free();
Add this to /arduino-0022/libraries/Ethernet/client.cpp. I put this below the Client::available() function.
int Client::free() {
if (_sock != MAX_SOCK_NUM)
return W5100.getTXFreeSize(_sock);
return 0;
}
If I were you, I would insert this into your present code after a transmit buffer write just as a check. Open the serial monitor, then try the connection from your phone. See if you get a "Low buffer" message. Put Serial.begin(9600) in setup.
I would expect the the library/w5100 to wait if the buffer is full. This is TCP...
I would too, but I do not see anywhere the buffer is protected from transmit buffer overflow. The TCP part doesn't start until you can get the data into the transmit buffer. The W5100 chip takes care of the rest of the TCP stuff from there.
It's a bit confusing...
In theory the W5100 can't declare any part of the buffer 'free' until it receives ACKs from the client. The ACKs can arrive in any order so in theory the buffer can become free in random order. I'm not sure how the chip handles this.
I've just been reading the datasheet for the W5100 and it doesn't mention any details.
The chip does have a "transmission retry" register which the Arduino library doesn't set. Maybe that's it.
I tried your fix ... it definitely made the connections last longer but it didn't fix it 100%. More fiddling is needed.
After reading the w5100 datasheet this morning I'm thinking it might be better to cut out the middle layer and drive the chip directly. It looks really easy to program and you can do a lot more stuff with it that way instead of being restricted to the Ethernet library.
I use a persistent TCP port 8085 connection with an Arduino as a client and a Linux box programmed as a "server". I have maintained a connection for three hours with it, but it is a localnet computer.
ADD: I did not mention this, but you must be careful how you print or write to the client.
This code sends 4 bytes in one packet
SurferTim:
I use a persistent TCP port 8085 connection with an Arduino as a client and a Linux box programmed as a "server". I have maintained a connection for three hours with it, but it is a localnet computer.
Yep, I've had no problems with that. I've been using the ethernet port to send debug output for quite a while with no problems (much better than continually opening/closing the serial console BTW).
I can also send files to local machines with no problem.
The problem only happens when I try to load web pages via my mobile phone. The connection runs for about a second then locks up in Client::write().
SurferTim:
How do other remote computers do with your code? Is it only the phone or do you experience the hang with any remote computer?
Don't know...I haven't got access to any.
The main thing is: The phone works fine if I put a delay between calls to Client.write(), it hangs if I don't.
Given that, I don't really see how it can be anything but the Ethernet driver.
SurferTim:
Would you mind posting the section of code that does the client.write?
Sure:
void FatFile::sendTo(Client& c)
{
if (ok()) {
FatFile::size_type r = remaining();
int sectorCount=0, packetCount=0;
const int chunkSize=128;
while (c.connected() and (r > 0)) {
// Get the next file sector into memory
FatFile::size_type s = clusterToSector(cluster)+sectorCount;
sdCard.getSector(s);
// Send the sector
int sectorOffset = 0;
while ((r > 0) and (sectorOffset < BYTES_PER_SECTOR)) {
FatFile::size_type t = r;
if (t > chunkSize) {
t = chunkSize;
}
c.write(&sdCard.sectorBuffer()[sectorOffset],t);
sectorOffset += chunkSize;
r -= t;
// It works if I put this delay here...
//delay(50);
}
if (++sectorCount == sectorsPerCluster) {
sectorCount = 0;
cluster = getNextCluster(cluster);
}
}
}
}
}
I'm not sure I understand this, but I am not familiar with the sd card yet:
c.write(&sdCard.sectorBuffer()[sectorOffset],t);
Have you checked the remaining transmit buffer size just before the delay(50)?
c.write(&sdCard.sectorBuffer()[sectorOffset],t);
sectorOffset += chunkSize;
r -= t;
// Serial.print("Buffer ");
Serial.println(c.free(),DEC);
// It works if I put this delay here...
//delay(50);
EDIT: I commented out the "Buffer" send. The serial port will have enough trouble keeping up once the buffer is full. You might even consider turning up the speed above the 9600 norm.
Now it will print only lines of numbers. Those will be the free bytes remaining in the transmit buffer after each write.
SurferTim:
I'm not sure I understand this, but I am not familiar with the sd card yet:
c.write(&sdCard.sectorBuffer()[sectorOffset],t);
I'm using my own SD card reader...sdCard.sectorBuffer() returns a pointer to 512 bytes of data, ie. the currently loaded disk sector.
SurferTim:
Have you checked the remaining transmit buffer size just before the delay(50)?
The only thing that seems to fix the problem is waiting for the buffer to be completely empty before sending (or a big delay - which is probably the same thing).
SurferTim:
Did you check the remaining transmit buffer size just before the delay(50)?
What was the result?
Try it with and without the delay.
It's not that easy to do: The Serial.print() is also a delay... :~
SurferTim:
ADD: Also check it from the localnet computer. The buffer method may be much faster with it than using the delay.
Of course...I only put in the delay to try and figure out what the problem is. Hopefully I can get rid of all the delays.
I'm almost finished writing my own W5100 controller now. That seems to be the way to go rather than trying to hack through the three layers of code in the Arduino Ethernet library (Client/socket/W5100).
I already ended up writing my own SD card driver, FAT16/FAT32, and HTTP code...what's an extra Ethernet driver on top of all that?
Of course...I only put in the delay to try and figure out what the problem is. Hopefully I can get rid of all the delays.
Do you know what the problem is?
I think it is due to bandwidth throttling applied by your ISP or the phone company. If that is the case, you will never get rid of the delays. Your goal will be to make the delay as short as possible.
Yes, but there is no TCP protocols on the transmit buffer input.
When the TCP connection says "no more for now", the output stops, and nothing is leaving the buffer. But you keep pounding the data into that buffer. It will overflow according to my calculations.
SurferTim:
When the TCP connection says "no more for now", the output stops, and nothing is leaving the buffer. But you keep pounding the data into that buffer. It will overflow according to my calculations.
Surely the W5100 will say "Stop!" until the data leaves the buffer...
I spent a bit of time investigating this today with packet sniffers and stuff to see what's going on in the TCP.
I seems that the free space register freaks out whenever a packet gets resent...it can return really crazy values - 20k free!
I'm not sure how to detect this yet. I've noticed that the status register goes to '1' when it happens but nowhere in the datasheet is this mentioned and none of the Wiznet sample code seems to check for it.
The datasheet for the w5100 shows that as the register (Sn_TX_FSR datasheet page 33) to check before writing to the transmit buffer. The function I used that was already in the w5100 library does multiple checks until it gets two returns from those functions that are the same value.
The only thing I can think of that might cause a problem, which it did for me, was the function that reads the 16 bit values from the w5100 registers. On mine, it affected the value returned by Client::available(). It would return much higher values that it should. I can't think of anything else that would affect that value outside the ethernet shield.
Edit: The modification also goes by the recommendation in the datasheet. Those two values are supposed to be read high byte first, then immediately low byte, or errors could occur.
