Go Down

Topic: Standalone Datacollecting Webserver (Read 9280 times) previous topic - next topic

tjgerow

Hello,

I have been searching the web for a while trying to find some direction on how to proceed with a project I am working on.

I have an arduino board with ethernet shield (webserver) reading temperature data and logging it to an SD card.  The Ethernet shield is connected to a wifi router using an RJ45 cable.  The wifi router is used to create a local wifi network so someone in the area with a mobile device (android, iphone, tablet, laptop,etc.) can connect to the network and access the Arduino webserver.

This whole setup is in a remote location with no cell coverage or internet access. 

The user can use a mobile wifi device with a browser to access the Arduino webserver and see current temperatures.  They can also go to webpages to calibrate the sensors and change other settings like logging frequency. 

Another feature I need to add is the ability to download the log files from the SD card through the HTML interface.  I am working on that now and have found lots of examples.

It is not ideal to have someone remove the SD card to get the data.  Thus the use of a wifi network.


Where I need help is the next step:  How can I graph data in HTML so that the user can see historical temperatures on their device?

I have found a bunch of examples using javascript but they all reference the javascript files on a different server connected to the internet.  This project has to be self contained and will not be able to access remote servers.

Is there a way to embed a graphing javascript/applet in the arduino webserver?

I have also found examples of installing DD-WRT firmware on the router and then using the router to host .css and .js files.  I really would like to avoid modifying the router firmware, unless that is the best option.

This recent topic:http://arduino.cc/forum/index.php/topic,143431.0.html is sort of similar and has a reference to a weather station project: http://arduino.cc/forum/index.php/topic,129122.0.html that is close to what I am trying to do (nice graphs).  I just don't have any internet access to call the javascript functions to do the graphs.

I need some direction, or a reality check to let me know this can not be done with just an Arduino, Ethernet Shield and a wifi router.

Thanks,
Tom

draythomp

There's a couple of reasons we use javascripts off of other servers.  The biggest being we don't want to write them ourselves since there are so darn many out there already, and the other big reason is that they are freaking huge.  However, if you follow the source trail to the actual code that puts up the graph, you can certainly do it.  The path I would use is to get one script at a time, these things seem to call script, after script, after script, modifying it to work locally on a laptop.  You WILL eventually get to the end of the various scripts and have a fully working set of code on the laptop. Then, you have to somehow shoehorn it onto the arduino.

Since you're already using an SD card, you may be able to do this pretty well, but it might take some time to get it all off the card and out over the air.

The possible problem with this is the license on the code.  Be sure to take a look at the various controls the authors have put on the pieces.  It's very likely you won't have any problem, but be sure to check this out best you can.
Trying to keep my house under control http://www.desert-home.com/

tjgerow

draythomp,

Thanks for the idea.  That will be my weekend project, to track down the actual scripts for graphing data.  I haven't spent a lot of time actually looking for the javascript source code because I was hoping there was already a library out there I could use.

Thanks again,
Tom

#3
Jan 26, 2013, 12:57 am Last Edit: Jan 26, 2013, 02:29 am by rockwallaby Reason: 1
Tom,

I first noticed your post in the thread by Llukkari and replied there, but now reading this post I understand a little better your setup.

I understand the Arduino is in the forest and is not connected to the outside world in any way?
But a person can go up to the weather station which has WiFi capabilities and can connect to it this way.

I am guessing you will be only wanting certain authorised personnel having access to the weather station?

I have a thought, but it needs more thinking,
It should be possible to store on the laptop that goes to site, all the needed html and javascript and css files.
This could be relatively easy if your laptop uses linux, such as Ubuntu, or if you have Mac laptops as these have built in web servers already to go.
I am unsure how you do this if you use that confounded other well known OS, which will remain nameless.

To do this you would need to configure your laptop as a server, so essentially you would serve your own resources.
Again, I haven't done this, but I think such an idea might work.
It would be very fast.

You want graphing?
I use highcharts, I like it as it is very customisable and able to represent your data very nicely.
I am currently playing around with this on my site, and am looking to having many more features for displaying data as I do in my industrial SCADA systems.

On my system (in my sig and under development), the Arduino is both a server and a client as well as running UDP to get correct time from a time sever.
All the content is now hosted off the Arduino, but earlier I had it being served from the Arduino.
Keep us posted.

Kind regards,
Paul

draythomp

There may well be complete, transportable charting software somewhere, I just haven't been able to find it.  I have managed to put together large chunks of the stuff, but never all of it.  Since I'm not limited as you are to not having web access, I went the other way and put everything out on the web so my little tiny arduino could do pretty sophisticated graphs and displays on the client browser.

As you progress through the various code, the development tools in Chrome could help you.  If you do an 'inspect element' (right click on the screen to get the menu) you can find the various sites that the code grabs from, then looking there, you can find the next one, etc.  It may turn out to be easier than I think and only a couple of things are needed to get the job done.

Have fun.
Trying to keep my house under control http://www.desert-home.com/

Quote
Dave wrote:
There may well be complete, transportable charting software somewhere, I just haven't been able to find it.


Hi Dave, have you checked out Highcharts at all?
http://www.highcharts.com/

They have two forms of charting software, one being highcharts and the other is highstocks, take your pick.
You don't need to reference or access their servers to be able to produce trends, they make their code available as a library, so you can host it yourself.

The chart you have on your blog site Dave looks to be the same as what highstocks looks like.

They have extra modules which allow you to export and to print the chart.
I'd say that was pretty transportable.

Regards,
Paul

draythomp

No, I have not.  Yes, now I have  XD

I haven't downloaded their stuff yet to play with, but I certainly will.  I first went and looked at their license and prices.  They let you have it free for non-commercial work; that fits my stuff exactly. 

tjgerow, this may be your ticket.  It's all in one place and could answer your problems if you can find a way to stuff it on an arduino.

rockwallaby, Thank you very much.

Trying to keep my house under control http://www.desert-home.com/

#7
Jan 26, 2013, 04:30 am Last Edit: Jan 26, 2013, 04:35 am by rockwallaby Reason: 1
This is where Zoomcat comes in with a nice bit of code that allows you to serve these files directly off the SD card on the Arduino, Zoomcat......

maybe this works
http://arduino.cc/forum/index.php/topic,129122.msg1089066.html#msg1089066

Paul

draythomp

Spoke too soon.  When I actually got to trying the charts out, they need supporting libraries from other sites.  I was several layers into other sites and finally gave up.  As far as their appearance, they are really nice and they can be dynamic; meaning that they can be made to change as new data is available.  They support tooltips that expand data when you hover over the data points, zooming in on certain areas, and many other nice features. 

I just don't see how they can work well with an isolated arduino in the middle of nowhere.

One notable thing was that the forum and example they provide didn't have a clear example of how to actually get data from some repository and display it on the screen as a graph.  Yes, they have examples, and offer them up to be examined by jsFiddle, but the dynamic and repository examples don't work in jsFiddle (at least I couldn't get them to)

Nice set of tools if you want to spend a week or two diving into the API and experimenting to get it right, but they don't seem to fit the bill for way far away devices that aren't big web servers.
Trying to keep my house under control http://www.desert-home.com/

Hi,
I was wondering if you found the solution to this i am also having this same problem have sensors reading in and displayed on the server just need to get it graphed and logged

Quote
ArduinoGreenHouse wrote:
have sensors reading in and displayed on the server just need to get it graphed and logged


Firstly, welcome to Arduino land.
When you say you have sensors reading and displayed on the server, I'm having trouble understanding what server are you displaying on?
Can you describe better your setup, are you using an external host site as your server, or the Arduino as server.

On what do you wish to graph, a computer screen, such as in a web browser or some other application?

Paul

#11
Feb 26, 2013, 12:20 am Last Edit: Feb 26, 2013, 12:26 am by ArduinoGreenHouse Reason: 1
Well at the moment i am using the Arduino to display the server, but have been looking on the form and thinking of using an external host, My plan is to have the sensors graphed in real time on the server with a datalogger with a RTC, my goal would be to have my server to have graphics like this example http://www.cs.helsinki.fi/u/ljlukkar/weatherstation
with 3/4 sensors been on the graph

Nick_Pyner

#12
Feb 26, 2013, 12:29 am Last Edit: Feb 26, 2013, 12:31 am by Nick_Pyner Reason: 1

Well at the moment i am using the Arduino to display the server, but have been looking on the form and thinking of using an external host, My plan is to have the sensors graphed in real time on the server with a datalogger with a RTC,



Sounds like a job for cosm?

https://cosm.com/feeds/83152

You don't actually need an RTC to do it, but it is still a good idea to have it for backup.

OK, I understand better now, thanks.

I took a lot of the concepts and ideas from Lasse Lukkari from an earlier project he worked on, and yes, it I think it looks good.
So, initially I had real time data being displayed on such a graph, but quickly came to the realisation that it had a few issues for me.
If you use such a configuration, that is, just displaying real-time data, you don't get to see the past data, so, no historical data is graphed.
And when you are displaying the real-time data, it all looks nice, but if you leave the page or re-fresh the page, it starts from a clean slate again.

Work out if that is all you need to do.
Or do you prefer to be able to look back through the data and graph it?
You can try it out by looking at mysetup, go to http://paulalting.com/hydrosolar/hydrosolar.php# and go to the 'trend' page.

It uses the very same graphing library as the weather station from Lasse Lukkari.

Quote
Nick wrote:
sOUNDS LIKE AJOB FOR COSM


Yes, you could do that, but you are dependant on that service and I think rather limited in how you can display your data.
If you have no web host site, it would be an alternative.

But in reality, you can also store the data onboard the Arduino on SD card, and then when a client connects, have the data sent to the client and displayed in a graphing program.

Right then, you better show your code (use the code tags).
Just to get an idea of your level of understanding;
Do you have some understanding and experience with in coding Arduino and web languages?

Describe your project in more detail;
List your hardware;

  • What Arduino board are you using

  • What ethernet chipset have you got

  • What type of sensors are you using



Your software;

  • Have you got a hosted site you can use

  • At what stage is your program as it is now



I am still in the middle of developing and many more features are to come yet, also for trending which is at a very basic stage at present.
I can look at getting my code online somehow, there's a bundle of Arduino files as well as files for server and the client side.
Or I can post some relevant sections of the code in a new topic maybe?

For the graphing, go to http://www.highcharts.com/ and have a play around with their demos, this is what is used.

I hope you will have some patience  8)

Go Up