Problem downloading files > 12K over Wifi

I'm using a brand-new Arduino Wifi shield to try to download a fairly large XML file (about 90K) from a web server. My board is a brand-new Arduino Uno.

The problem is that the connection consistently closes after reading around

The problem isn't my wifi network, as I can open a telnet session from a laptop on the same network, mimic exactly the same HTTP headers and watch the entire file download

I thought that maybe I wasn't processing the data fast enough, so I removed all processing from my loop() function to get:

void loop() 
{
  if( client.connected() )
  {
    if( client.available() > 0 )
    {
      Serial.print("Connected: ");
      Serial.println(client.available());
    }
    
    while(client.available())
    {
      int chunkLength = client.readBytes(buffer, 128);
      charsRead += chunkLength;
      Serial.print("Read chunk: ");
      Serial.print(chunkLength);
      Serial.print(" towards total of: ");
      Serial.println(charsRead);
    }
  }
}

The last thing this outputs tends to be in the neighborhood of 12 or 14K.

I also tried a variant of the above where I just read one char at a time instead of reading a buffer of 128 characters - I got the same result.

Anyone know what's going on here? I've tried with multiple servers too, so this doesn't seem to be a server-specific problem. Is there any way to get error information out of the wifi module? I'm not able to tell specifically what went wrong - all I know is that I can't download any more data.

Thanks!

What are you doing with what you are 'downloading'?. An Uno has a very limited amount of ram (2K) and 12K simply isn't going to fit in a 2K ram space. A better answer will depend on your code that you neglected to post.

Bob

Bob, my code is included in the post - in my original code, I'm scanning a large RSS feed for two characters of information that I want to display on an LCD screen. But since I had problems with that, I stripped it down to the simplest possible bit of code - I just read through what I get and discard it. But even that code doesn't work.

I really just need a very small part of the feed, so I only need to keep a few bytes of the data in memory at a time.

I apologize for the "Code Comment", I was interrupted in trying to answer your questions. In reading your snippet of code it looks like an adding machine.. while(client.available()) { int chunkLength = client.readBytes(buffer, 128); charsRead += chunkLength; " The only thing I see this doing is incrementing chunkLength... I don't see anything being done with the data in the code you posted and without more of your real code, It'd hard to tell what the real issue is. I would still wonder about ram use.

Bob

Like I said, I stripped out all my logic to see if the Arduino could read the full file without trouble, and without the complication of code not related to just reading the stream. So all this snippet does is count the characters read. But even this simple bit of code - which never stores more than 128 bytes of the stream at a time - fails. I also tried reading one character at a time and reading in 512-byte chunks.

I would expect that this code would run with charsRead somewhere close to 90 * 1024. If that worked, then I could add my parsing logic back. But this makes me worry that there's some fundamental problem with downloading files over wifi.

What is charsRead defined as? Maybe worth working with longs or unsigned longs instead of ints as the int goes negative after 32767

It's a long. It's not an overflow problem - it prints charsRead on each round through the cycle, so I can see it increasing in 128-byte increments until it comes to a stop at 12K.

I contacted Arduino about this, and it's a known issue that they have a fix for, though it's not publicly available yet.

They created this Github issue to track the problem. My understanding is that this is just a software fix, and once the next version of the code is out, you can just update the wifishield code and your shield will work properly.