Downloading files > 20kb doesn't work UPDATE: it must be WiFi issue...

Hey,

i try to dowload a file > 16kb from a webserver - connected with the wifi shield (UNO Rev3) but i always got a download abort before the file download finished.
If i read the length of the file and make client.read() while i < the content length the file gets filled to the original file size but the content
is not correct (just some of the first bytes are the same).
Maybe its too much for SPI to download and write the same time files larger than 15kb??
Smaller files ~4kb work fine - writing large files to the sdcard without wifi connection work fine too (thanks to fat16lib and his tools).
Dowloading the file with an arduino ethernet work without problems too.

So is anybody out there with the same problem? Would it be a solution to stop download at a specific size and resume it at this point? Is this possible with web hosted files?

I use the current WiFi and SdFat lib.

Thanks in advance!!
-andreas

AndreasW79:
If i read the length of the file and make client.read() while i < the content length the file gets filled to the original file size but the content is not correct (just some of the first bytes are the same).

Do you check for client.available() before reading a byte? If not, you may be reading faster than the server can send.

johnwasser:
Do you check for client.available() before reading a byte? If not, you may be reading faster than the server can send.

The read is within a while client.available loop.

Little update - i tried to output the file to serial.write, same here :confused:
Perhaps its not an SD Card issue rather than a WiFi one?
I also disabled SPI for SD slot with

pinMode(4, OUTPUT);
digitalWrite(4, HIGH);

Tomorrow i will try the same sketch with an ethernet Shield UNO R3 and an Arduino Ethernet and come
back with some new information!

So far -thanks.

-andreas

Back again...

i tried the following with 3 different Boards:

Arduino UNO R3 w/WiFi Shield (official)

#include <SPI.h>
#include <WiFi.h>

#define debug(a) Serial.print(millis()); Serial.print(": "); Serial.println(a);
#define debug2(a, b) Serial.print(millis()); Serial.print(": "); Serial.print(a); Serial.println(b);

void setup(){
  Serial.begin(9600);

char ssid[] = "ssid";     //  your network SSID (name)
char pass[] = "pass";    // your network password

//pinMode(4, OUTPUT);
//digitalWrite(4, HIGH);

// -- Initialize WiFi connection
  int status = WL_IDLE_STATUS;
  status = WiFi.begin(ssid, pass);
  if (status != WL_CONNECTED) {
    Serial.println("No WiFi Connection!");
  }

  // print your local IP address:
  debug2("IP address: ", WiFi.localIP());
}
WiFiClient client;
char* host = "192.168.2.222";
int port = 80;

int totalRequestSize = 0;
int failures = 0;
int requests = 0;

void loop() {
  if (client.connect(host, port)) {
    debug("Connected.");
    client.println("GET /test.file HTTP/1.0");
    client.println("Host: 192.168.2.222");
    client.println();
    int length = 0;
    while(client.connected()) {
      debug("Still connected");
      // the server will close the connection when it is finished sending packets
      while(client.available()) {
        // ...but not until this buffer is empty
        Serial.write(client.read());
        length++;
      }
      debug("Done reading for now...");
    }
    // close your end after the server closes its end
    client.stop();
    
    requests++;
    if (length != totalRequestSize) {
      failures++;
    }
    debug2("Total read: ", length);
    Serial.print("Total failures: "); Serial.print(failures); Serial.print("/"); Serial.println(requests);
  } else {
    debug("Couldn't connect.");
  }
  delay(10000);
}

Arduino UNO R3 w/Ethernet Shield (official)

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0x90, 0xA2, 0xDA, 0x0D, 0x4C, 0x75 }; //physical mac address UNO R3 EthernetShield
EthernetClient client;

#define debug(a) Serial.print(millis()); Serial.print(": "); Serial.println(a);
#define debug2(a, b) Serial.print(millis()); Serial.print(": "); Serial.print(a); Serial.println(b);

void setup(){
  Serial.begin(9600);

  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    debug("DHCP Failed");
    // no point in carrying on, so do nothing forevermore:
    while(true);
  }
  // print your local IP address:
  debug2("IP address: ", Ethernet.localIP());
}

char* host = "192.168.2.222";
int port = 80;

int totalRequestSize = 0;
int failures = 0;
int requests = 0;

void loop() {
  if (client.connect(host, port)) {
    debug("Connected.");
    client.println("GET /test.file HTTP/1.0");
    client.println("Host: 192.168.2.222.net");
    client.println();
    int length = 0;

    while(client.connected()) {
      debug("Still connected");
      // the server will close the connection when it is finished sending packets
      while(client.available()) {
        // ...but not until this buffer is empty
        Serial.write(client.read());
        length++;
      }
      debug("Done reading for now...");
    }
    // close your end after the server closes its end
    client.stop();
    
    requests++;
    if (length != totalRequestSize) {
      failures++;
    }
    debug2("Total read: ", length);
    Serial.print("Total failures: "); Serial.print(failures); Serial.print("/"); Serial.println(requests);
  } else {
    debug("Couldn't connect.");
  }
  delay(1000);
}

Arduino Ethernet(official)

same as above with different MAC

Testfile

[url]http://pastebin.com/DKbStKG8[/url]

Results:

UNO R3 w/WiFi Shield

Total read: 14467 - missing bytes (any loop)

Last two lines from Serial Output:
3AAAAAABBBBBBBCCCCCCDDDDDDDDEEEEEEFFFFFFGGGGGHHHHHHIIIIIIJJJJJJJKKKKKKKLLLLLLLMMMMMMNNNNNOOOOOPPPPPPQQQQQQRRRRSSSSSTTTTTUUUUUVVVVVWWWWWXXXXXYYYYYYZZZZZ
EOF EOF

The last line is available but between some lines gets lost....

[b][u]With [/u][/b]
pinMode(4, OUTPUT);
digitalWrite(4, HIGH);

7AAAAAABBBBBBBCCCCCCDDDDDDDDEEEEEEEFFFFFFGGGGGHHHHHHIIIIIIJJJJJJJKKKKKKKLLLLLLLMMMMMMNNNNNOOOOOPPPPPPQQQQQQRRRRSSSSSTTTTTUUUUUVVVVVWWWWWXXXXXYYYYYYZZZZZ
8AAAAAABBBBBBBCCCCCCDDDDDDDDEEEEEEEFFFFFFGGGGGHHHHHHIIIIIIJJJJJJJKKKKKKKLLLLLLLMMMMMMNNNNNOOOOOPPPPPPQQQQQQRRRRSSSSSTTTTTUUUUUVVEEEEEEFFFFFFGGGGGHHHHHHIIIIIIJJJJJJJKKKKKKKLLLLLLLMMMMMMNNNNNOOOOOPPPPPPQQQQQQRRRRSSSSSTTTTTUUUUUVVVVVWWWWWXXXXXYYYYYYZZZZZ
EOF EOF
Total read: 13955

UNO R3 w/Etnerhet Shield

Total read: 27779 - complete expected length (any loop)

Arduino Ethernet

Total read: 27779 - complete expected length (any loop)

Summary:

What i can exclude:

-WiFi Network ( I tried different router an different networks (Netgear, AVM, Cisco))
-UNO R3 (as it works with the Ethernet Shield without any problem)
-SD Card (as i disabled it and doesnt include any SD libraray)
-Content Server (as it works with the Arduino Ethernet and UNO R3 w/Ethernet Shield)

Conclusion:

There must be something wrong with the WiFi Connection / WiFi Library,
so i hope my debugging can lead someone to an solution?
Maybe i do something wrong or missed something else?!

Hopefully someone can lead me to the right direction!

Thanks a lot!

-andreas

... can some mod pls move my topic to Networking??

Thanks

Ugh, I'm having a very similar problem. I'm not able to read more than 12K or so, though the same exact HTTP headers work fine from my laptop running on the same wifi network.

Did you resolve this issue?

I contacted Arduino about this, and it's a known issue that they have a fix for, though it's not publicly available yet.

They created this Github issue to track the problem. My understanding is that this is just a software fix, and once the next version of the code is out, you can just update the wifishield code and your shield will work properly.