Standalone Datacollecting Webserver

Hello,

I have been searching the web for a while trying to find some direction on how to proceed with a project I am working on.

I have an arduino board with ethernet shield (webserver) reading temperature data and logging it to an SD card. The Ethernet shield is connected to a wifi router using an RJ45 cable. The wifi router is used to create a local wifi network so someone in the area with a mobile device (android, iphone, tablet, laptop,etc.) can connect to the network and access the Arduino webserver.

This whole setup is in a remote location with no cell coverage or internet access.

The user can use a mobile wifi device with a browser to access the Arduino webserver and see current temperatures. They can also go to webpages to calibrate the sensors and change other settings like logging frequency.

Another feature I need to add is the ability to download the log files from the SD card through the HTML interface. I am working on that now and have found lots of examples.

It is not ideal to have someone remove the SD card to get the data. Thus the use of a wifi network.

Where I need help is the next step: How can I graph data in HTML so that the user can see historical temperatures on their device?

I have found a bunch of examples using javascript but they all reference the javascript files on a different server connected to the internet. This project has to be self contained and will not be able to access remote servers.

Is there a way to embed a graphing javascript/applet in the arduino webserver?

I have also found examples of installing DD-WRT firmware on the router and then using the router to host .css and .js files. I really would like to avoid modifying the router firmware, unless that is the best option.

This recent topic:http://arduino.cc/forum/index.php/topic,143431.0.html is sort of similar and has a reference to a weather station project: http://arduino.cc/forum/index.php/topic,129122.0.html that is close to what I am trying to do (nice graphs). I just don't have any internet access to call the javascript functions to do the graphs.

I need some direction, or a reality check to let me know this can not be done with just an Arduino, Ethernet Shield and a wifi router.

Thanks, Tom

There's a couple of reasons we use javascripts off of other servers. The biggest being we don't want to write them ourselves since there are so darn many out there already, and the other big reason is that they are freaking huge. However, if you follow the source trail to the actual code that puts up the graph, you can certainly do it. The path I would use is to get one script at a time, these things seem to call script, after script, after script, modifying it to work locally on a laptop. You WILL eventually get to the end of the various scripts and have a fully working set of code on the laptop. Then, you have to somehow shoehorn it onto the arduino.

Since you're already using an SD card, you may be able to do this pretty well, but it might take some time to get it all off the card and out over the air.

The possible problem with this is the license on the code. Be sure to take a look at the various controls the authors have put on the pieces. It's very likely you won't have any problem, but be sure to check this out best you can.

draythomp,

Thanks for the idea. That will be my weekend project, to track down the actual scripts for graphing data. I haven't spent a lot of time actually looking for the javascript source code because I was hoping there was already a library out there I could use.

Thanks again, Tom

Tom,

I first noticed your post in the thread by Llukkari and replied there, but now reading this post I understand a little better your setup.

I understand the Arduino is in the forest and is not connected to the outside world in any way? But a person can go up to the weather station which has WiFi capabilities and can connect to it this way.

I am guessing you will be only wanting certain authorised personnel having access to the weather station?

I have a thought, but it needs more thinking, It should be possible to store on the laptop that goes to site, all the needed html and javascript and css files. This could be relatively easy if your laptop uses linux, such as Ubuntu, or if you have Mac laptops as these have built in web servers already to go. I am unsure how you do this if you use that confounded other well known OS, which will remain nameless.

To do this you would need to configure your laptop as a server, so essentially you would serve your own resources. Again, I haven't done this, but I think such an idea might work. It would be very fast.

You want graphing? I use highcharts, I like it as it is very customisable and able to represent your data very nicely. I am currently playing around with this on my site, and am looking to having many more features for displaying data as I do in my industrial SCADA systems.

On my system (in my sig and under development), the Arduino is both a server and a client as well as running UDP to get correct time from a time sever. All the content is now hosted off the Arduino, but earlier I had it being served from the Arduino. Keep us posted.

Kind regards, Paul

There may well be complete, transportable charting software somewhere, I just haven't been able to find it. I have managed to put together large chunks of the stuff, but never all of it. Since I'm not limited as you are to not having web access, I went the other way and put everything out on the web so my little tiny arduino could do pretty sophisticated graphs and displays on the client browser.

As you progress through the various code, the development tools in Chrome could help you. If you do an 'inspect element' (right click on the screen to get the menu) you can find the various sites that the code grabs from, then looking there, you can find the next one, etc. It may turn out to be easier than I think and only a couple of things are needed to get the job done.

Have fun.

Dave wrote: There may well be complete, transportable charting software somewhere, I just haven't been able to find it.

Hi Dave, have you checked out Highcharts at all? http://www.highcharts.com/

They have two forms of charting software, one being highcharts and the other is highstocks, take your pick. You don't need to reference or access their servers to be able to produce trends, they make their code available as a library, so you can host it yourself.

The chart you have on your blog site Dave looks to be the same as what highstocks looks like.

They have extra modules which allow you to export and to print the chart. I'd say that was pretty transportable.

Regards, Paul

No, I have not. Yes, now I have XD

I haven't downloaded their stuff yet to play with, but I certainly will. I first went and looked at their license and prices. They let you have it free for non-commercial work; that fits my stuff exactly.

tjgerow, this may be your ticket. It's all in one place and could answer your problems if you can find a way to stuff it on an arduino.

rockwallaby, Thank you very much.

This is where Zoomcat comes in with a nice bit of code that allows you to serve these files directly off the SD card on the Arduino, Zoomcat......

maybe this works http://arduino.cc/forum/index.php/topic,129122.msg1089066.html#msg1089066

Paul

Spoke too soon. When I actually got to trying the charts out, they need supporting libraries from other sites. I was several layers into other sites and finally gave up. As far as their appearance, they are really nice and they can be dynamic; meaning that they can be made to change as new data is available. They support tooltips that expand data when you hover over the data points, zooming in on certain areas, and many other nice features.

I just don't see how they can work well with an isolated arduino in the middle of nowhere.

One notable thing was that the forum and example they provide didn't have a clear example of how to actually get data from some repository and display it on the screen as a graph. Yes, they have examples, and offer them up to be examined by jsFiddle, but the dynamic and repository examples don't work in jsFiddle (at least I couldn't get them to)

Nice set of tools if you want to spend a week or two diving into the API and experimenting to get it right, but they don't seem to fit the bill for way far away devices that aren't big web servers.

Hi, I was wondering if you found the solution to this i am also having this same problem have sensors reading in and displayed on the server just need to get it graphed and logged

ArduinoGreenHouse wrote: have sensors reading in and displayed on the server just need to get it graphed and logged

Firstly, welcome to Arduino land. When you say you have sensors reading and displayed on the server, I'm having trouble understanding what server are you displaying on? Can you describe better your setup, are you using an external host site as your server, or the Arduino as server.

On what do you wish to graph, a computer screen, such as in a web browser or some other application?

Paul

Well at the moment i am using the Arduino to display the server, but have been looking on the form and thinking of using an external host, My plan is to have the sensors graphed in real time on the server with a datalogger with a RTC, my goal would be to have my server to have graphics like this example http://www.cs.helsinki.fi/u/ljlukkar/weatherstation
with 3/4 sensors been on the graph

ArduinoGreenHouse: Well at the moment i am using the Arduino to display the server, but have been looking on the form and thinking of using an external host, My plan is to have the sensors graphed in real time on the server with a datalogger with a RTC,

Sounds like a job for cosm?

https://cosm.com/feeds/83152

You don't actually need an RTC to do it, but it is still a good idea to have it for backup.

OK, I understand better now, thanks.

I took a lot of the concepts and ideas from Lasse Lukkari from an earlier project he worked on, and yes, it I think it looks good. So, initially I had real time data being displayed on such a graph, but quickly came to the realisation that it had a few issues for me. If you use such a configuration, that is, just displaying real-time data, you don't get to see the past data, so, no historical data is graphed. And when you are displaying the real-time data, it all looks nice, but if you leave the page or re-fresh the page, it starts from a clean slate again.

Work out if that is all you need to do. Or do you prefer to be able to look back through the data and graph it? You can try it out by looking at mysetup, go to http://paulalting.com/hydrosolar/hydrosolar.php# and go to the 'trend' page.

It uses the very same graphing library as the weather station from Lasse Lukkari.

Nick wrote: sOUNDS LIKE AJOB FOR COSM

Yes, you could do that, but you are dependant on that service and I think rather limited in how you can display your data. If you have no web host site, it would be an alternative.

But in reality, you can also store the data onboard the Arduino on SD card, and then when a client connects, have the data sent to the client and displayed in a graphing program.

Right then, you better show your code (use the code tags). Just to get an idea of your level of understanding; Do you have some understanding and experience with in coding Arduino and web languages?

Describe your project in more detail; List your hardware;

  • What Arduino board are you using
  • What ethernet chipset have you got
  • What type of sensors are you using

Your software;

  • Have you got a hosted site you can use
  • At what stage is your program as it is now

I am still in the middle of developing and many more features are to come yet, also for trending which is at a very basic stage at present. I can look at getting my code online somehow, there's a bundle of Arduino files as well as files for server and the client side. Or I can post some relevant sections of the code in a new topic maybe?

For the graphing, go to http://www.highcharts.com/ and have a play around with their demos, this is what is used.

I hope you will have some patience 8)

I have done a few projects with Arduino and also know C and know a Bit of web development

I am using the Arudino mega with the Ethernet shield the sensors I have are thermistor LDR and a Humidity sensor

Yea I have a hosting site not much done on it! So far i have the 3 sensors reading the values and printing them on the Ethernet server

oh thats sound cool

Yea I have looked into that site for graphing seems good

Thanks

ArduinoGreenHouse, sorry for the delay getting back to you, I have been working on the project some more, getting the solar current sensor electronics installed and configured in the program. Hopefully that will start to show values on the overview page as well as the trend page when the sun comes up in the morning Australian time. For the next few days I unfortunately will not have a chance to do anything in the way of presenting the software as I will be away, though will do at the first opportunity.

Sounds like you have the right hardware bits for this to work, especially need the Mega with the extra memory as the program is too big for the smaller processors. It might be that you have no need for sections of my code, such as the PID control loop for turbine and for solar dumping of excess energy.

If you are using the Arduino IDE I need to confirm that you can simply use my files from Eclipse IDE, which are filename.cpp and filename.h for the header, by renaming them? If anybody following this thread knows, then it would be good to know.

Paul

Hi, thanks for replying yea ok thats fine, yea I have the right hardware and yea i wouldn't think i need them sections of that code, basically my server will show a nice graphical webpage with a graph that shows the 3 sensors over a period of time which can be saved in a certain time intervals.

If you are using the Arduino IDE I need to confirm that you can simply use my files from Eclipse IDE, which are filename.cpp and filename.h for the header, by renaming them?

I not sure what you mean by this :/

ArduinoGrenHouse wrote in reply:

If you are using the Arduino IDE I need to confirm that you can simply use my files from Eclipse IDE, which are filename.cpp and filename.h for the header, by renaming them?

I not sure what you mean by this :/

I use Eclipse as my IDE for all my programming, arduino, html, javascript and so on. I haven't used the Arduino IDE for some time now. My question is more about confirming that the files I use in Eclipse, you can directly use in the Arduino IDE if that is what you use. I think you may need to change the file extension of cpp to ino, i.e. 'main.cpp' to main.ino' for it to work in Arduino IDE. Can anybody else confirm this? If you use Eclipse, then that is great.

I can't attach the files in these posts as some of them exceed the file size that can be use on the forum. What I propose, is to use dropbox as the file storage location.

To get you started, I have copied the main section of code, where the setup and main loop are. It's not so big. You need to look at them and start to see if you can make sense of them, they are documented. Then as you start to understand the various components and can successfully compile it you should be right to make the necessary chamges for your setup. I am not going to strip stuff out that you may not need, that will be up to you.

The dropbox location is https://www.dropbox.com/sh/gtjv0k6sos6yuf5/EKzPXRT4zj?m

I will continue to clean up the other files over the coming days and pop them into dropbox as well. Let me know if you can access the two files.

Start a new project and check that you have the libraries listed at the top of main.cpp. They are Ethernet, EthernetUDP, TimerThree (from the Arduino playground) as well you will need Webduino library from its Github location, https://github.com/sirleech/Webduino.

Paul

Hi, Just out of curiosity, I have the hosting sever set up, uploaded basic HTML website , Just wondering couldn't I use the arduino to send the data of the three singals to pachube and get pachube to graph and send the graph onto my site? Was looking into this and I guess I found some example of this a setting up a trigger of some sort, can this be done? if so is it anygood/reliable, would cut my time alot!

Ok will give that a shot, have and great looking website now will try with eclipse!