Hanging when getting larger files

I’m trying to use a yun shield to download a bmp file which I’m saving to SD card and then to display on a 2.8" tft. If I download a 20k file there is no problem however if I try a 240k file the get hangs.

For example this file fails


however this one works


I’ve tried to use client.get as well as client.getAsynchronously

Is there any way to download large files using a yun shield?

Alternatively can someone suggest a low cost wifi option that this will work with? So far I’m on my 3rd different wifi device and nothing in this space seems to be able to stream larger files.

Ok that’s weird, it just started working. Hmm now to find out what I did

Ok looks like if I do a get it hangs however if I do a getAsynchronously and sit in a look getting data as it comes in then I get it all be it very slowly.

Alternatively can someone suggest a low cost wifi option that this will work with? So far I’m on my 3rd different wifi device and nothing in this space seems to be able to stream larger files.

Your problem could very well be memory and not the WiFi device. You don’t say which board you’re using with the shield, but most Arduino processors have 2 to 2.5 kbytes of memory, so you have to be careful to read the file in very small chunks. It’s impossible to say what’s actually going on in your case without knowing more details and seeing your code, but the fact that you are now on your third WiFi device and still having trouble tells me that the problem is not in the WiFi device, but in your code.

Unlike most WiFi interfaces, the Yun has the advantage of having a rather powerful Linux computer running the networking. I would suggest writing a Python (or other language) script that runs on the Linux side, and which fetches the file. This will be much faster and more reliable since the Linux side runs much faster than the Arduino processor, and it has plenty of memory. Then once received, the script sends the image to the Arduino sketch in small fixed size chunks so that the sketch can send it to the display. You can still run the Linux script using Process.runAsynchronously().

Ok looks like if I do a get it hangs

That makes sense: get() won’t return until the process is complete. However you don’t have enough memory to be able to read the whole thing in one batch, so the process will never complete until you read some of the data, but with run() you can’t read any data until the process completes. Catch-22.

The answer is to use runAsynchronously(). Doing as much as possible on the Linux side should speed things up, but you are still faced with sending that 240k of data over the serial port that is between the two processors. This will take some time any way you look at it. A large file like that is going to take 20 seconds at a minimum just to send all of the bytes over the serial port, and it will actually take longer because there is some data transmission overhead when talking between the two processors.

You’re running into the memory and speed limitations of the Arduino processors. They are small and limited capacity. If you want to dramatically speed things up, you need to move to a much larger and faster processor, one which can drive the display directly without going through such a small processor. You might be better of with something like a Raspberry Pi or BeagleBone?

Yeah aware of the memory issues which is why I'm reading and writing direct to SD with little buffering. I was assuming that the get fetched the file into the yun's memory and then when I did the client.read it was bringing it down byte by byte to the arduino mega which I was passing directly off to SD. In fact even the 20k file that I got successfully with the .get should have been too much for the Arduino which is why I though the yun itself was buffering for me.

I'll be looking at doing a few hundred units so unit price is more of an issue than speed at the moment so trying to stay with something as cheap as possible. But for a prototype I think this might work.

I just noticed the saving to SD card part of this, I missed it this morning. I’m assuming you’re talking about the SD card on the Yun Shield? If so, you can really speed things up by moving as much of the logic as possible to the Linux side.

I also just noticed that you are mentioning client.get(), which I read as process.get(). That’ll teach me to not read things carefully when I’m on my tablet…

It sounds like your general program flow might be:

  • In the sketch, use an HttpClient object to get() the image file
  • Read the image file byte by byte using client.read() and then…
  • … write the image file byte by byte to the SD card using the FileIO class.
  • Finally, read the SD card file using the FileIO class, and write it to the LCD screen.

Is this what you’re doing? If so, the serial bottleneck between the two processors that I mentioned is really killing you, as you are sending that image file data over the serial link THREE times: once as you get the image (Linux is actually getting the image over the network, and then sending it to the sketch) then again as you write to the SD card (you’re actually sending the file back down to Linux so it can write it to the SD card) and then once again to read from the SD card (Linux is actually reading the file and sending it back up to the sketch.)

If that’s what you’re doing, you can get three times the performance by writing a Linux script to get the image file and save it directly to SD card, and then using FileIO in the sketch to read the SD card. Or, just skip the SD card and go from the get call (either using HttPClient like you are, or using a Process class runAsynchronously() call to run curl as I initially assumed you were doing) and then write directly to the LCD.

I’m sure there are way to speed up what you’re doing and making more efficient. Seeing your current code would allow us to give you some suggestions. Just keep in mind that the Yun Shield is NOT the same thing as a typical WiFi shield and an SD card shield. The Yun has a Linux processor, and the SD card and all networking (WiFi and Ethernet) are managed by the Linux processor, not the sketch. It’s important to realize what is going on, and what actions cause data to be sent over the serial port between processors. Not being aware of this, and thinking like a normal WiFi Shield and SD card shield can cause a lot of wasted processing.