Pages: [1] 2   Go Down
Author Topic: Ethernet hangs/doesn't send  (Read 3775 times)
0 Members and 1 Guest are viewing this topic.
Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm writing a web server for my Ethernet Arduino to serve files from the SD card...it's looking pretty good.

It works fine when I access files from machines on the local network but if I try to access them across the Internet on my mobile phone the file sends hang after a few blocks of data have been sent.

The send loop is very simple, just read the file and send it in chunks of 64 bytes using a "Client" object.

I can "fix" the problem by putting in a delay between each block send, eg. "delay(50)". If I do that it all works 100% perfectly (although *very* slowly).

I'm starting to suspect there's a problem in the Ethernet library when the Arduino sends data faster than the other end can receive it. Does anybody know anything about this? Has anybody seen anything like this before?
Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 147
Posts: 6038
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Sounds like you are overflowing the ethernet transmit buffer.

I just finished a function for the ethernet library to avoid this problem. It allows you to check the free space in the transmit buffer. I have been testing it somewhat like the client.available() function. I call it client.free(). It returns the size of the unused transmit buffer.

I have been using it like this:

Code:
if(client.free() > 1000)
{
   client.print("test");
}
It could be optimized by determining the size of your data and checking if there is that much space remaining rather than just using a set number.

Would you be willing to test it? It is only a few lines of code in two files. I'll post the additions here if you are interested.

« Last Edit: September 11, 2011, 08:54:40 pm by SurferTim » Logged

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Sounds like you are overflowing the ethernet transmit buffer.

I would expect the the library/w5100 to wait if the buffer is full. This is TCP...

I just finished a function for the ethernet library to avoid this problem. It allows you to check the free space in the transmit buffer. I have been testing it somewhat like the client.available() function. I call it client.free(). It returns the size of the unused transmit buffer.

I was thinking of doing something similar today. The Ethernet library has no non-blocking options at the moment...it definitely needs some.

But sure, I'll try it. Post the code!

Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 147
Posts: 6038
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I would expect the the library/w5100 to wait if the buffer is full. This is TCP...
I would too, but I do not see anywhere the buffer is protected from transmit buffer overflow. The TCP part doesn't start until you can get the data into the transmit buffer. The W5100 chip takes care of the rest of the TCP stuff from there.

Add this to /arduino-0022/libraries/Ethernet/client.h. I put it below the declaration for available().
Code:
virtual int free();
 

Add this to /arduino-0022/libraries/Ethernet/client.cpp. I put this below the Client::available() function.
Code:
int Client::free() {
  if (_sock != MAX_SOCK_NUM)
    return W5100.getTXFreeSize(_sock);
  return 0;
}

If I were you, I would insert this into your present code after a transmit buffer write just as a check. Open the serial monitor, then try the connection from your phone. See if you get a "Low buffer" message. Put Serial.begin(9600) in setup.
Code:
int newFree = client.free();

if(newFree < 1000)
{
   Serial.print("Low buffer ");
   Serial.println(newFree, DEC);
}

With the transmit buffer empty, the function returns 2048.
« Last Edit: September 12, 2011, 06:18:29 am by SurferTim » Logged

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I would expect the the library/w5100 to wait if the buffer is full. This is TCP...
I would too, but I do not see anywhere the buffer is protected from transmit buffer overflow. The TCP part doesn't start until you can get the data into the transmit buffer. The W5100 chip takes care of the rest of the TCP stuff from there.

It's a bit confusing...

In theory the W5100 can't declare any part of the buffer 'free' until it receives ACKs from the client. The ACKs can arrive in any order so in theory the buffer can become free in random order. I'm not sure how the chip handles this.

I've just been reading the datasheet for the W5100 and it doesn't mention any details.

The chip does have a "transmission retry" register which the Arduino library doesn't set. Maybe that's it.


I tried your fix ... it definitely made the connections last longer but it didn't fix it 100%. More fiddling is needed.

After reading the w5100 datasheet this morning I'm thinking it might be better to cut out the middle layer and drive the chip directly. It looks really easy to program and you can do a lot more stuff with it that way instead of being restricted to the Ethernet library.

Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 147
Posts: 6038
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

If you refer to the Retry Count Register, this is how it is set.
http://arduino.cc/forum/index.php/topic,71905.0.html
The default is 8.

I use a persistent TCP port 8085 connection with an Arduino as a client and a Linux box programmed as a "server". I have maintained a connection for three hours with it, but it is a localnet computer.

ADD: I did not mention this, but you must be careful how you print or write to the client.
This code sends 4 bytes in one packet
Code:
byte outBuffer[4] = {65,66,67,68};
client.write(outBuffer,4);
This code sends 4 bytes each in its own packet
Code:
byte outBuffer[4] = {65,66,67,68};
client.write(outBuffer[0]);
client.write(outBuffer[1]);
client.write(outBuffer[2]);
client.write(outBuffer[3]);
Same apparently applies to the print and println functions. Each write, print, or println call generates its own packet.

« Last Edit: September 12, 2011, 12:50:08 pm by SurferTim » Logged

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

If you refer to the Retry Count Register, this is how it is set.
http://arduino.cc/forum/index.php/topic,71905.0.html
The default is 8.

I guess the library just uses the default then...

I use a persistent TCP port 8085 connection with an Arduino as a client and a Linux box programmed as a "server". I have maintained a connection for three hours with it, but it is a localnet computer.

Yep, I've had no problems with that. I've been using the ethernet port to send debug output for quite a while with no problems (much better than continually opening/closing the serial console BTW).

I can also send files to local machines with no problem.

The problem only happens when I try to load web pages via my mobile phone. The connection runs for about a second then locks up in Client::write().

Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 147
Posts: 6038
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

How do other remote computers do with your code? Is it only the phone or do you experience the hang with any remote computer?
 
Would you mind posting the section of code that does the client.write?
Logged

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

How do other remote computers do with your code? Is it only the phone or do you experience the hang with any remote computer?
 
Don't know...I haven't got access to any.

The main thing is: The phone works fine if I put a delay between calls to Client.write(), it hangs if I don't.
Given that, I don't really see how it can be anything but the Ethernet driver.

Would you mind posting the section of code that does the client.write?

Sure:

Code:
void FatFile::sendTo(Client& c)
{
  if (ok()) {
      FatFile::size_type r = remaining();
      int sectorCount=0, packetCount=0;
      const int chunkSize=128;
      while (c.connected() and (r > 0)) {
        // Get the next file sector into memory
        FatFile::size_type s = clusterToSector(cluster)+sectorCount;
        sdCard.getSector(s);
        // Send the sector
        int sectorOffset = 0;
        while ((r > 0) and (sectorOffset < BYTES_PER_SECTOR)) {
          FatFile::size_type t = r;
          if (t > chunkSize) {
            t = chunkSize;
          }
          c.write(&sdCard.sectorBuffer()[sectorOffset],t);
          sectorOffset += chunkSize;
          r -= t;
          // It works if I put this delay here...
          //delay(50);
        }
        if (++sectorCount == sectorsPerCluster) {
          sectorCount = 0;
          cluster = getNextCluster(cluster);
        }
      }
    }
  }
}

Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 147
Posts: 6038
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm not sure I understand this, but I am not familiar with the sd card yet:
Code:
c.write(&sdCard.sectorBuffer()[sectorOffset],t);

Have you checked the remaining transmit buffer size just before the delay(50)?

Code:
         c.write(&sdCard.sectorBuffer()[sectorOffset],t);
          sectorOffset += chunkSize;
          r -= t;
 
          // Serial.print("Buffer ");
          Serial.println(c.free(),DEC);

          // It works if I put this delay here...
          //delay(50);

EDIT: I commented out the "Buffer" send. The serial port will have enough trouble keeping up once the buffer is full. You might even consider turning up the speed above the 9600 norm.

Now it will print only lines of numbers. Those will be the free bytes remaining in the transmit buffer after each write.
« Last Edit: September 13, 2011, 09:36:27 am by SurferTim » Logged

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm not sure I understand this, but I am not familiar with the sd card yet:
Code:
c.write(&sdCard.sectorBuffer()[sectorOffset],t);


I'm using my own SD card reader...sdCard.sectorBuffer() returns a pointer to 512 bytes of data, ie. the currently loaded disk sector.


Have you checked the remaining transmit buffer size just before the delay(50)?

The only thing that seems to fix the problem is waiting for the buffer to be completely empty before sending (or a big delay - which is probably the same thing).

Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 147
Posts: 6038
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Did you check the remaining transmit buffer size just before the delay(50)?
What was the result?
Try it with and without the delay.

I am curious about the numbers. Does the buffer size decrease to less than 128 without the delay?

ADD: Also check it from the localnet computer. The buffer method may be much faster with it than using the delay.
« Last Edit: September 13, 2011, 02:32:52 pm by SurferTim » Logged

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Did you check the remaining transmit buffer size just before the delay(50)?
What was the result?
Try it with and without the delay.

It's not that easy to do: The Serial.print() is also a delay...  smiley-confuse

ADD: Also check it from the localnet computer. The buffer method may be much faster with it than using the delay.

Of course...I only put in the delay to try and figure out what the problem is. Hopefully I can get rid of all the delays.

I'm almost finished writing my own W5100 controller now. That seems to be the way to go rather than trying to hack through the three layers of code in the Arduino Ethernet library (Client/socket/W5100).

I already ended up writing my own SD card driver, FAT16/FAT32, and HTTP code...what's an extra Ethernet driver on top of all that?

Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 147
Posts: 6038
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Of course...I only put in the delay to try and figure out what the problem is. Hopefully I can get rid of all the delays.
Do you know what the problem is?

I think it is due to bandwidth throttling applied by your ISP or the phone company. If that is the case, you will never get rid of the delays. Your goal will be to make the delay as short as possible.

« Last Edit: September 14, 2011, 05:16:10 am by SurferTim » Logged

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5522
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Of course...I only put in the delay to try and figure out what the problem is. Hopefully I can get rid of all the delays.
Do you know what the problem is?

Not yet. I want to bypass the Ethernet code and see if it's the library or the W5100.

PS: I'm suspecting the Ethernet library at the moment, the W5100 is claimed to be mature and suitable for use in medical equipment.

I think it is due to bandwidth throttling applied by your ISP or the phone company.

That's what the TCP part of TCP/IP is for - to make sure all the packets arrive....!

Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Pages: [1] 2   Go Up
Jump to: