An HTTP Server for Arduino Mega with file upload/download and AJAX capabilities

As part of another project, I needed an HTTP server able to upload/download files to/from the SD-card and serve AJAX requests from common browsers.

I tried Webduino but had problems in file upload, so I decided to tailor a new one, based on the experience done with Webduino, but more reliable and stable for my applications.

Now I have extracted the sources as an independent library called HttpSvr. As it is, it has been tested only on Arduino Mega and the Ethernet Shield, so it is not yet ready for being a real general library, but I hope it might be useful to the community for further developments and fun.

HttpSvr can be found at Google Code Archive - Long-term storage for Google Code Project Hosting. as a Google Code Project.

It is a spare-time project, so I cannot assure professional follow-up, but I will be very glad to have comments, opinions and contributions aimed to improve it.

Thanks in advance,
eca

Hi.
good work.
but, possible rewrite code for uno?

Yes, definitely! And thanks for your interest!
Actually the porting to UNO it is in my short-term plans, and I am already considering how to lower memory occupation both for program and data, that seems to be the biggest showstopper now.
Suggestions and contributions are definitely welcome, while I will try to post updates on the forum as soon as available.

Cordially,
eca

After several experiments with HttpSvr on Arduino UNO, and after several modifications to the library that have lowered the memory footprint especially on RAM (both .data and .bss), I could run the sample code on UNO.

Incoming client connections are managed correctly, but the amount of RAM left for stack and other user data is really, really small: less than 300 bytes - not enough to run functions like "sendResFile", i.e. the one that reads a resource file from SD card and sends it to the client, or to manage file upload. For what concerns program memory, the sample program (without LCD library) takes about 27 KBytes of program memory over 32 available, in my opinion leaving too little space for other user software.

A possible solution to reduce memory occupation both for program and data space is to eliminate SD library, but in this configuration the HttpSvr library itself makes really little sense, because only dynamic pages can be managed - something that you can already do using the official Ethernet library.

All in all, I think that HttpSvr makes sense only if it can give you something more than official libraries: simple access to pages on SD card, file upload and response to AJAX requests. But this costs a bit more computational power and is intended to improve your potential, not reduce it by leaving you too little RAM or Flash.

So, I think I'll use HttpSvr as it is only on MEGA and not on UNO, even if I think I'll use parts of the source code also in other projects for UNO that require HTTP connections, as hopely others will do.

HTH
eca

I find this very interesting, i like to serve AJAX pages from a measurement project.

I would like to use your web server with the official WiFi shield, i have previously been able to modify webdruino for the WiFi shield but this is to complex for me to understand, i can't find any references to ethernet.h.

Would it be possible without major changes to get it to work with WiFi? if so, i would love to get some hints how to.

/N

Hello nekstrand,

Thank you for your interest in HttpSvr.

HttpSvr has actually taken some ideas out of Webduino, especially the possibility to bind an URL to a callback function providing dynamic pages. But, while Webduino leverages the original Ethernet library for driving the W5100, I chose to rewrite the low-level API to the ethernet chip because of problems in uploading files to the SD card. With this new hand-made driver, my problems have disappeared.

So, HttpSvr comes in two separate levels: one is the Http Server itself, and the other is the W5100 driver. In theory - and hopely in practice, too - porting HttpSvr to WiFi should be a matter of replacing the driver, i.e. the W5100xx.yyy files in the utility folder, yet keeping the API unchanged and leaving other things unchanged too. I have no experience with the WiFi shield, but my basic idea was exactly to separate the two levels so as to allow seamless replacement of the low level - provided that the API remains more or less the same.

I think that you can start from having a look to the utility folder and see what is in there. I tried to put a lot of comments there, however feel encouraged to ask anything you might want to know and is not clear in the source. You are definitely welcome!

Cordially,
eca

Thanks for your reply,

I think that is to hard with my very limited knowledge, but i will at least have a look at it

/N

A tutorial that might be of interest.

Thanks zoomkat,

I actually skipped the WiFi shield, it seems a bit unstable, now i use the ethernet shield with an Apple airport express i had lying around as a wifi bridge.

enrico_ecarduino, would it be possible to use the Ethernet / EthernetUdp libraries in parallel with the Httpsvr, i use the EthernetUdp to set the time from a NTP-server (which lets me run real time without a RTC).

Hello nekstrand,

Not 100% sure they can work together, because HttpSvr bypasses the Ethernet library and manages the W5100 chip and its sockets independently. So I suspect they would clash if used together. But...

But I am working right in these days on a project that would extend the HttpSvr library with an UDP client (and an HttpClient, and a DNS client...), exactly to get time via NTP. If you think you can wait, say, one week, I can post the entire project for NTP time on this forum or make it available on Google Code. I would be extremely glad to have you check it out, in case it might be of your interest.

One note about NTP: someone on the internet suggests not to be too pedantic in requesting time from NTP servers, because they tend to block excessively frequent requests coming from the same IP. Rather, the suggested approach is to ask for NTP timestamp once at startup, then go with your internal timer or RTC.

HTH,
Cordially,
eca

Wow, that would be really great.

A DNS client is a super update as well, then it would be possible to use the NTP pool's as well.

Its absolutely enough to make a NTP request at startup and then it is probably enough once a week, but i think i like to check it daily.

Hi nekstrand,

Yestarday I have added a branch to HttpSvr-arduino code (Google Code Archive - Long-term storage for Google Code Project Hosting.) where a modified (improved?) version of HttpSvr library with management of UDP datagrams has been added.

A modified version of the HttpMega example is also included, that uses UDP for connecting to a NTP server on startup and presenting the resulting timestamp string in the first HTML page "index.html" via AJAX request. This example works for me, but please consider the following:

  • This branch is kind of experimental, so do not cargo-cult it!
  • I had some problems while importing the new code in svn repository, so I did it almost manually. After 1.5 hours spent importing files one by one and restarting RapidSVN every single time :astonished:, I realized that I missed one directory below "examples". So the correct directory hierarchy is: "branches/examples/HttpMega/[files]" and not "branches/examples/[files]". Sorry for that, please keep it in mind if you plan to download the content of "branches": you should add a directory "HttpMega" manually under "examples", then move files from "examples" down one level in "HttpMega".

The DNS client is not stable yet, so I think it is not safe to include it, I would like to revise the code a bit more before releasing it in the wild.

Hope I'll be back soon with a more polished version of the library, however I hope you (and others) could take some inspiration from what is already done.

Cordially,
eca

I have downloaded it and will try it out, i'll be back.

/N

I'm getting the server part to work as previously, but the ntp info just informs me "bad datagram", have not figured out why yet.

I will also try to get it to work with the time library, this takes care of some of the code and offers quite some functionality.

Update, I'm using #undef HTTPSVR_USE_LCD.

/N

I would try to understand if the internet access is correctly working first. Double check if your gateway/subnet mask configuration is correct.

Secondly, I suggest you to monitor network traffic from/to Arduino with Wireshark (http://www.wireshark.org/) so as to understand if the "bad datagram" is the request or the response. Wireshark also gives you the possibility to analyze packets at different levels so as to understand precisely what is going on in the communication.

HTTPSVR_USE_LCD should only work as an on/off switch for the LCD part (library and sketch code) and should not affect network functionality. Just to be extra-sure, you may try with and without it defined, and see if there is any difference in behavior. What is expected is just a reduced code size and no output on LCD when #undefined.

In the meanwhile, I am trying to find some time to work further on the library, so as to make it more robust and error tolerant. A better diagnostic and error recovery capability would be desirable - and is what I would like to do as the next step.
If you find errors, or make some improvements, please share on the forum, I'll be glad to consider them in future releases!

Thanks for your contribution,
eca