Arduino + Ethernet Shield - web cache

Hi all,

I'm working on a project, for witch I need to connect to internet, download one HTML page (~12K), store it on the SD card, and use it for further processing. It's little tricky to download the page and store it to the SD, because I need not stop to stop/start the W5100 and the SD card. As you know, they can't operate simultaneously. The other problem is that the content is too big for the Arduino RAM (~12K) so I need to store it on portions. I was thinking also to add one EEPROM chip and to buffer it there before saving to the SD card, but I read in the forum that there are some problems when trying to work with SD.h and Wire.h together. Any ideas, clues or advices how to proceed. And if someone has expiriense with working both with ethernet and writing to SD, I'll be glad, if he comment.

P.S. This is my first post, so HELLO WORLD!

When you add in all the additional shields and components you need, wouldn't it be simpler to move to a platform that has enough memory and handles network access in a less dodgy way than the Arduino?


Can you strip out some of the content on the page to make it small enough to store in ram till the sd card is ready? A lot of the html would be redundant depending on what you want out of the 12k page. Or maybe even pre process it to get the info you need then discard the rest. Both of these methods would have to be done on the stream as it came in.

Or alternatively is there an I2c or serial storage device with more storage?