Arduino - Processing - Web Interface - Historical Data Trending

I am relatively new to using Arduino. I am looking for anyone with experience using arduino basically as a datalogger for the various analog/digital I/O. I am trying to design a small scale anaerobic digestion / solar hybrid system (off-grid) using arduino as the brains. The end goal is to provide a web interface where a user could view historic trends of the various process parameters and make changes to system operation inside same GUI. I am thinking of using MySQL, Apache Web Server on Ubuntu, Processing, and Arduino together to accomplish this. I am unsure of how exactly these things would work together.

Maybe the MySQL database is unnecessary and everything could be stored on the SD card of the ethernet shield? I am inexperienced in the web-server aspect of this task and I am looking for a little guidance or recommendation.

So you want to build the following "pipeline":

sensors --> Arduino --> ethernetshield --> PHPform --> MYSQL DB --> presentation --> user

A good place to start us to add a LCD to the Arduino to show locally the values of the sensors. That is

sensors --> Arduino --> LCD

if that works start with designing the database (be prepared for multiple Arduino's) and the presentation of the data

Then Add the PHP form to fill the database, initially manually.

Then add the ethernetshield to fill the phpform automatically.

An alternative is to look at pachube as this has automated the back end ...

So is Processing taken out of the loop in the “pipeline” you suggested? I have never programmed in PHP and I have limited experience in Java.
1st pipeline - sensors → Arduino → ethernetShield → PHPform → MySQL DB–> presentation → user
How is the presentation portion of the pipeline created? Does the PHPform create the HTML page that the user would use as an interface?

I looked at pachtube (I think it’s called Cosm now?). I am confused exactly what this service could provide. You feed this a data stream and it will act as a data storage/retrieval server? Or is the interface created in Cosm? I would like the capability for full customization in the user interface which is why I have been leaning towards using Processing for the GUI. The aspect I struggle with is how to push this interface to the web and allow for remote viewing/control. I was also thinking of uploading the Firmata program to Arduino and using Processing to interface with Arduino directly. Is it possible for the pipeline to work like this?:

sensors → Arduino → ethernetShield → Processing <–> MySQL DB (stored on laptop connected to arduino)
|–> Apache Web Server → Java Web App

Would you have to purchase your own domain to accomplish this? Sorry if this idea is convoluted and impossible. Thank you for the advice. Much appreciated.

I've done something similar, using C# instead of Processing and SqlServer instead of MySQL (though those changes are trivial).

Storing the data in the database is better in terms of long term storage and the ability to perform trend analysis. If it is stored on the Arduino, you need to take the Arduino out of action to retrieve the SD card, to get its data.

Then, you still need to perform the analysis on the data, which will typically involve reformatting the data anyway.

Storing the data in a database can support automatic time stamping, meaning that there is one less thing that the Arduino needs to deal with (sending time/date information from a RTC).

I'm not sure that Processing can access MySQL directly, though it might be possible to develop a library to accomplish this. PHP certainly can.

PHP is ideal for developing web applications, so rather than having Processing do the visualization part, let the web browser to that.

You could use a data logger with SD card pretty cheap (under $20 in parts) and sneaker-net the SD to a computer once a day/week/month if instant data is not required.

@Wishiwasanindian where are you located? I'm working on something similar in NorCal. have you made any progress? We should connect