Best way to do Datalogging?

Hi i've builded a DustDuino with a Shinyei PPD42 sensor, now it prints the values with softwareserial library to an HC05 Bluetooth module.

I receive those values on a tablet with a BT terminal app and save it manually to a .txt, i send the .txt to my laptop and import it in Excel to make the plot.

That is boring...

What is the best way to do data logging?

I've seen that Xively that was used in DustDuino project is now only for business application.

I would like to know how to approach this problem and buy the correct parts to finish my project, i think that do datalog to the sdcard is not the best way because is more complicated than now.

Ethernet Shield? WiFi with ESP module? Help me please :confused:

I use Parallax PLX-DAQ with an older version of Excel on a laptop to import data directly into a spreadsheet.

Data come in via the laptop serial port, which could be a Bluetooth or other type of wireless connection.

never heard about Parallax PLX-DAQ seems quite nice but i'm on a Mac o Linux machine.

i would like to make it working more standalone not always with a computer connected, server via wifi or ethernet i think that would be better but i've no idea of how to approach that

Nikka93:
Hi i've builded a DustDuino with a Shinyei PPD42 sensor, now it prints the values with softwareserial library to an HC05 Bluetooth module.

I receive those values on a tablet with a BT terminal app and save it manually to a .txt, i send the .txt to my laptop and import it in Excel to make the plot.

That is boring...

Rather less boring is to send the data to Bluetooth Graphics Terminal instead, and get the graph live on the phone.

I've seen that Xively that was used in DustDuino project is now only for business application.

I have only used Xively myself but there are alternatives, like Thingspeak and Grovestreams that people use.

I would like to know how to approach this problem and buy the correct parts to finish my project, i think that do datalog to the sdcard is not the best way because is more complicated than now.

Ethernet Shield? WiFi with ESP module?

If you are serious about what you are doing, you will keep recording to SD as a backup. The simplest way to go is just get an Ethernet shield, which invariably includes ans SD slot. If you want/need to use WiFi you can connect the Ethernet shield to a WiFi router/repeater. The combo is still cheaper than a WiFi shield, more versatile, and a lot easier to use. Arduino is not involved with the WiFi operation.

I find python useful for this...here is my code. Takes a string via Serial, splits it by comma and then stores values in a mySQL database, this is on a Raspberry Pi, but should work regardless on any platform:

import serial
import datetime
import time
import MySQLdb

#Get the data from serial//
port = serial.Serial("/dev/ttyAMA0",baudrate =  9600, timeout = 10.0)

#connect to mySQL DB rawdata

db = MySQLdb.connect('localhost',"user","password","rawdata")
curs=db.cursor()
counter=0
#Do this always
while True:
	datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
	#read the serial data in buffer (35 characters worth)
        rcv = port.readline(100)
	counter+=1
	
	#split the original string in to parts
	if counter>30:	
		try:	#Take the data and delimit it in to seperate arrays for each determinant.
			delimited_data = rcv.strip().split(',')
			
			#check to see if there is any issue with the LUX sensor:
			try:
				val = float(delimited_data[4])
			except:
				#make it so the data is read and the last LUX column made equal to zero
				#print("Error in the delim data.")
				
				
				curs.execute("INSERT INTO GHCdata (H1,H2,T1,T2,LUX) VALUES(%s,%s,%s,%s,0)", (delimited_data[0],delimited_data[1],delimited_data[2],delimited_data[3]))
				db.commit()
				counter=0;

			#If there are no issues with LUX sensor overload: Put the data in to the mysql table.
			curs.execute("INSERT INTO GHCdata (H1,H2,T1,T2,LUX) VALUES (%s,%s,%s,%s,%s)",(delimited_data[0],delimited_data[1],delimited_data[2],delimited_data[3],delimited_data[4]))
			db.commit()
			#print "Added data to mySQL"
			counter=0;

		#if there were any other errors (faulty whatever) then say there is an error and add it to a log file.
		except:
			#print("ERROR")
			f=open('data.txt','a')
			f.write("\n"+"Error - got :"+ ",".join(delimited_data)+" at "+ datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
			counter=0;

Johnny010:
I find python useful for this...here is my code. Takes a string via Serial, splits it by comma and then stores values in a mySQL database, this is on a Raspberry Pi, but should work regardless on any platform:

import serial

import datetime
import time
import MySQLdb

#Get the data from serial//
port = serial.Serial("/dev/ttyAMA0",baudrate =  9600, timeout = 10.0)

#connect to mySQL DB rawdata

db = MySQLdb.connect('localhost',"user","password","rawdata")
curs=db.cursor()
counter=0
#Do this always
while True:
datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
#read the serial data in buffer (35 characters worth)
        rcv = port.readline(100)
counter+=1

#split the original string in to parts
if counter>30:	
	try:	#Take the data and delimit it in to seperate arrays for each determinant.
		delimited_data = rcv.strip().split(',')
		
		#check to see if there is any issue with the LUX sensor:
		try:
			val = float(delimited_data[4])
		except:
			#make it so the data is read and the last LUX column made equal to zero
			#print("Error in the delim data.")
			
			
			curs.execute("INSERT INTO GHCdata (H1,H2,T1,T2,LUX) VALUES(%s,%s,%s,%s,0)", (delimited_data[0],delimited_data[1],delimited_data[2],delimited_data[3]))
			db.commit()
			counter=0;

		#If there are no issues with LUX sensor overload: Put the data in to the mysql table.
		curs.execute("INSERT INTO GHCdata (H1,H2,T1,T2,LUX) VALUES (%s,%s,%s,%s,%s)",(delimited_data[0],delimited_data[1],delimited_data[2],delimited_data[3],delimited_data[4]))
		db.commit()
		#print "Added data to mySQL"
		counter=0;

	#if there were any other errors (faulty whatever) then say there is an error and add it to a log file.
	except:
		#print("ERROR")
		f=open('data.txt','a')
		f.write("\n"+"Error - got :"+ ",".join(delimited_data)+" at "+ datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
		counter=0;

Thanks man for sharing your code i hope that it will be useful for more pro users than me! Or to me in the future!

Nick_Pyner:
Rather less boring is to send the data to Bluetooth Graphics Terminal instead, and get the graph live on the phone.

I have only used Xively myself but there are alternatives, like Thingspeak and Grovestreams that people use.

If you are serious about what you are doing, you will keep recording to SD as a backup. The simplest way to go is just get an Ethernet shield, which invariably includes ans SD slot. If you want/need to use WiFi you can connect the Ethernet shield to a WiFi router/repeater. The combo is still cheaper than a WiFi shield, more versatile, and a lot easier to use. Arduino is not involved with the WiFi operation.

In those days i read the specification of those two websites and i've a couple of answer to do:

Thingspeak limit for the data storage is only the last hour? The only API limit seems to be the 15 data send for second

API Rate Limit
The open service via ThingSpeak.com has a rate limit of an update per channel every 15 seconds. This limit is so that the service can remain free and give everyone a high-level of service.

The Grovestream limit for free users looks only Data I/O /Month 5 MBs
My files are only about 15Kb for day so i should stay in the limit and the data storage looks unlimited.

ThingSpeak seems to be more used with ESP WiFi module that would be the best for my use because my sensor should stay outdoor 24h/7d and would need only a USB power supply and not a wifi to ethernet repeater

Have you seen MegunoLink?

Have you seen?

.