Go Down

Topic: PC File Upload to Arduino SD Card through Browser and Ardy Ethernet (Read 473 times) previous topic - next topic

CatweazleNZ

Hi All

Does anyone have any interest or thoughts about this subject? My Arduino web server home automation application at http://www.2wg.co.nz is serving me well and a month or so back I implemented browser based file downloads from the SD card for a whole variety of file types.

But I also want to upload files from my PC to the Arduino SD card without shutting down my application and removing the SD card. That means a standard browser based file upload function through my Arduino's ethernet functionality to the SD card on the Arduino board.

I cannot find a solution on the WWW. I have researched the subject quite a bit, think I know the requirement and believe it is a do able (but complex) task.

I already have a partial implementation - but I am currently running low on RAM. I won't be able to complete the first operational implementation for a week or two until I do another round of maintenance, procedure, string and RAM optimisation.

I thought I would raise this now in case anyone has an interest in this and knows of a solution or has some ideas that I should be incorporating into my work.

Cheers

Catweazle NZ.



Don't know if 100% in topic, but you might want to have a look at http://forum.arduino.cc/index.php?topic=203450.0, where I proposed a possible solution that worked for me. Please note however that, due to memory requirements, it runs only on Mega, so I don't know if this implementation could be useful to you.

HTH,
Enrico

CatweazleNZ

That link was about the only reference I could find to this topic on the internet - but I have not seen the source code.

It seems to me that the key to this issue is understanding the format of html requests and building efficient parsers to extract relevant information.

I have two parsers - one to extract general html request data including field values posted from forms, cookies etc. That parser can also identify a multi-part form html request and a copy of it can be found at http://forum.arduino.cc/index.php?topic=264312.0

I am now working on a secondary parser to extract file content for text and binary files from the multi-part section of the html request. I think I have all of the foundation code in place and hope to be able to implement the file upload this weekend.

You can check out my application at http://www.2wg.co.nz. Within the directory http://www.2wg.co.nz/OTHER.DIR the file QBF.TXT was created by my file upload functionality in every respect except that the actual file content is not yet being extracted from the file upload html request. The downloadable files at http://2wg.co.nz/PUBLIC.DIR/ and http://2wg.co.nz/IMAGES.DIR/ (and others) are what I want to maintain using application file uploads without needing to shut down the application and remove the SD card.

Yes, memory is an issue. I use Freetronics Ethermega cards and am able to fit my total application (130KB - 8,000+ lines of code) into its 256KB of program space, 8KB of RAM and 4KB of EEPROM. After some maintenance I am currently running at about 1500 bytes minimum free RAM memory - that should be more than enough for the added file upload code which will be processed through a new 128 byte buffer to allow identification of the embedded multi-part form field boundaries.

Check out http://www.2wg.co.nz/34993/ - this web page displays real time memory usage statistics. The functionality has been of great value in helping me understand, minimise and manage my application's RAM usage through the last six months.

Cheers

Catweazle NZ



CatweazleNZ

#3
Sep 07, 2014, 02:10 am Last Edit: Sep 07, 2014, 04:51 am by CatweazleNZ Reason: 1
I have a first version of my file upload functionality in place. But for large files I only get about 1790 bytes. I suspect this is related to the size of the socket input buffer with excess file data being discarded.

I have seen references to the W5100 having circular buffers. Does anyone have any perspectives on this matter?

If you look here http://www.2wg.co.nz/UPLOAD.DIR/ you can see several of these truncated upload files.

Catweazle NZ

Hi Catweazle,

Obviously I can't say what is the reason of file truncation, however I can say that, when I developed the first versions of HttpSvr using the official Ethernet library and W5100 driver, I ran under the impression that something in the sequence of operations for reading incoming bytes was somehow obscure or not always predictable. This is why I decided to rewrite the whole stuff from scratch bypassing the Ethernet library - just to have full awareness of what was going on. Of course I don't mean that the library doesn't work, only that I could not find exactly what was wrong in my way of using it for file upload.
Anyway, the unsuccessful experiments of long file uploading made with the official library seemed to work successfully with my homemade version (up to several MB - beware, it takes long), so I didn't go any further in investigating the Ethernet library.
In my code I tried to put very special care in clearing W5100 flags and moving counters as prescribed in the datasheet, and it seems to work better for me.
Last but not least, I am not putting on HttpSvr project professional resources, so I did not perform serious test campaigns, nor am I in the condition of offering professional support to the project. However, if you are interested in having a look at the sources, you can find them at https://code.google.com/p/httpsvr-arduino/.

HTH,
Enrico

CatweazleNZ


Hi Catweazle,

Obviously I can't say what is the reason of file truncation, however I can say that, when I developed the first versions of HttpSvr using the official Ethernet library and W5100 driver, I ran under the impression that something in the sequence of operations for reading incoming bytes was somehow obscure or not always predictable. This is why I decided to rewrite the whole stuff from scratch bypassing the Ethernet library - just to have full awareness of what was going on. Of course I don't mean that the library doesn't work, only that I could not find exactly what was wrong in my way of using it for file upload.
Anyway, the unsuccessful experiments of long file uploading made with the official library seemed to work successfully with my homemade version (up to several MB - beware, it takes long), so I didn't go any further in investigating the Ethernet library.
In my code I tried to put very special care in clearing W5100 flags and moving counters as prescribed in the datasheet, and it seems to work better for me.
Last but not least, I am not putting on HttpSvr project professional resources, so I did not perform serious test campaigns, nor am I in the condition of offering professional support to the project. However, if you are interested in having a look at the sources, you can find them at https://code.google.com/p/httpsvr-arduino/.

HTH,
Enrico

Thanks for the comments and I will analyse your source code in due course. Success would be valuable to my application plans so I will pursue this matter and dig very deep in search of a solution.

Yes, I think the ethernetclient code is a bit problematic so I also expect to be bypassing it and invoking code directly from the socket and W5100 classes/source code files.

I am suspicious of the handshaking going on and want to research that. I have seen some references to the fact that the W5100 does not support PAUSE FRAME but may support ENQ/ACK. I want to find out what all that means and see if I can pin down any issue in there.

Given that this is an after hours/weekend activity and I have other priorities this issue may take me weeks if not a month or two to resolve. That is OK by me - in the meantime others may see this thread, give me a solution and save me a lot of effort (I hope).

I already have had to deal with stuck Ethernet sockets in my application. I do not know why they get stuck but my application is able to detect them and force a disconnection.

Catweazle NZ

CatweazleNZ

#6
Oct 02, 2014, 12:03 pm Last Edit: Oct 06, 2014, 10:05 am by CatweazleNZ Reason: 1
After a good debugging session tonight I isolated the cause of file upload failures which was occurring after say 1800 bytes (previously) and at lower values during tonight's testing.

It seems that the statement  "if (G_EthernetClient.available() == 0)" will return true part way through the file upload when one block of data has been processed and the system is waiting for the next block of data. So my software now enters into a waiting loop until the next block of data arrives when the file upload continues.

The upload process is not fast - I guess you cannot expect that with these processors.

When  "if (G_EthernetClient.available() == 0)" returns true for ten continuous seconds my application would assume no more data is coming and terminate the upload/processing. As that time the end of file boundary is also found (normally) so the process completes accurately.

If you look at the files in http://www.2WG.co.nz/UPLOAD.DIR/ you have examples of a set of files all successfully loaded through an Arduino webserver website onto an SD Card on my main Arduino system.

I expect to be testing and refining this functionality over the next couple of weeks and will then publish some code on my application website at http://www.2wg.co.nz/PUBLIC.DIR/

Cheers

Catweazle NZ

Hi Catweazle,

I am very interested in the upload functionality, and in your whole project generally.
I have checked you homepage, and I actually intend to implement some similar Web interface for my SD card manager and home automation.
I also offer my setup (custom board with ATmega128A + Enc28J60 + SD card) for testing if you would like to share some code with me.

Please keep me up to date with your developments.
Cheers.

Go Up