Data collection with a long delay

Oct 6th
This image please: I have a weather station. It takes in information, real-time. Every hour I want all the standing information put in a database. Then I want it to sleep for an hour. My project involves magnetic north. I want it to wake up every fifteen minutes, turn on the 4.5-volt laser, take a reading and go back to sleep. I want the data, actually, just one number, put into a database with the UNIX time. I use Python and sqlite3 and have used this system successfully with other projects.
The whole is working, to a first approximation, fine. But the database is picking up extra lines. The C++ is sending line breaks to the python and the Python is adding extra rows. Like 20 to 200 blank rows,
I am doing the delay with a ‘for loop’ and then a “delay(60000)” with every iteration. So, the i=i+1 value is the number of minutes. (60000 milliseconds in a minute)
I am pretty sure that the handling of the delay is the problem (not the Python). If I change the 60000 and the iteration of the ‘for loop’ the number of blank lines changes.
I tried a straight “delay(900000)” and the database picked up 298 blank lines.
Should I just go toward a millis() function?
I have included the C++ (I use a mega board because it has the extra digital inputs.)

//This is the delay platform
//my delay functions are not working correctly.  I am just now understanding it!!
//  this is also a practice platform  for the laser  relay (4.5V)

const int relayPin = 8;  //output...  at intervals, the laser is turned on

int i = 0;  //  for loop variable

void setup()
{

  pinMode (relayPin, OUTPUT);
  digitalWrite (relayPin,HIGH);  //  relay high is the default off;  normally open is open

  Serial.begin(9600);
  Serial.println("hello world i is 2 the deklay is 60k");  //this is a 2 minute daley  and my wake up for the python
  numberForPython = 777777;

}

void loop()    {
  Serial.println (numberForPython);        //this goes out to Python "start Python signal"
  numberForPython = (numberForPython + 1);  //  this goes into the database as a series of numbers
  delay (900);
/*
                          this is the delay
 *                         what is the interval?
 *                         
 *                         the '  i  ' in the 'for loop' is the number of minutes 
                           and the 'delay' is the 60 thousand milliseconds for one minute
  */

  for (i = 0 ; i < 10; i++) {
    delay(60000);                    // this would be a ten minute delay
  }
/*  this is the locaion of the 40 LDR sensors.  I have it as 40 'if' statements.
 *   only one should go positive and that is sent out to Python and put in a
 *   database.  
 *   
 *   THE PROBLEM IS THAT THE ARDUINO IS SENDING OUT LINE BREAKS THAT GO INTO (Python) THE 
 *   DATABASE AS EMPTY ROWS WITH JUST THE UNIX TIME.
 *   
 *   If I modify the above 'if' loop the number of rows changes. I can find no relationship.
 *   Elliminating the 'if' statement (just a big 'delay()') seems to produce more lines!   
 */
  digitalWrite (relayPin, LOW);    //  low is on ..  green light on   turn on the laser
  //the green light should come on  (module JQ3CF- 05vDC works fine)
  // for a few seconds LASER ON


  delay(1000);
  digitalWrite(relayPin, HIGH);       //turns off the laser  HIGH is off  (module JQ3CF- 05vDC works fine)
  Serial.println (numberForPython);   //goes out to python


}

This code doesn't compile. Where is the definition of numberForPython?

How is this connected to a database? All it does is send numberForPython twice every hour.

Are you sure the board isn't resetting? How are this laser and relay powered?

Please post code that compiles without errors, successfully loads and demonstrates the problem.

How does the result of Serial.print() get sent to a database?

Yes, instead of blocking, you skip..

Using just a call to delay in a for loop, i would be worried the compiler optimization might have something to say about that..

Track your millis and only execute when needed but keep your loop looping..

good luck.. ~q

I have moved to a different approach and still have the same problem. This is using the millis() function. Compare time now to time plus increment. If true then do it, otherwise wait. I had a bit of trouble with it. I am more careful with the code. (My development computer is not my regular computer)
The sqlite3 is way easier than mysql. The Python requires the sqlite3 library. I assume that this requires the sqlite3 software. The Python puts the database in the same folder, at least the way I have it configured. With the Arduino running, kill the serial monitor before starting the Python. I do not remember this issue with other projects. If you download this code and can run it I would be amazed. Advanced beginners, like me find the system part of the project terrifying.
////////////////////////////////////////

//This is the delay platform
//my delay functions are not working correctly.  I am just now understanding it!!
//  this is also a practice platform  for the laser 4.5 volt relay

const int relayPin = 8;  //output...  at intervals, the laser is turned on

long  numberForPython = 8888;
long now_time = 340000 ; //  just pick a value
long old_time  =  2000;  //  must be lower than 340000

///////////////////////////////////////
void setup()
{

  pinMode (relayPin, OUTPUT);
  digitalWrite (relayPin, HIGH); //  relay HIGH is the default off;  'normally open' is default according to how I have it wired

  Serial.begin(9600);
  Serial.println("now time and old time.  30000 delay");  //this is a 30 second delay my wake up for the python
  numberForPython = 887777;
  Serial.println (now_time);
  Serial.println (old_time);

}
/////////////////////////////////////////
void loop()    {
  //Serial.println (numberForPython);        //this goes out to Python and  this is my  "start Python" signa
  numberForPython = (numberForPython + 1);  //  this goes into the database as a series of numbers
                                           //  This gives us the number of iterations
  //now_time = millis();
  delay(400);
  //Serial.println (now_time);
  if (now_time > old_time) {


   
    digitalWrite (relayPin, LOW);    //  low is on ..  green light on   turn on the laser
    //the green light should come on  (module JQ3CF- 05vDC works fine)
    // for a few seconds LASER ON
 /*  this is the locaion of the 40 LDR sensors.  I have it as 40 'if' statements.
         only one should go positive and that is sent out to Python and put in a
         database.
        */
    delay(1000);
    digitalWrite(relayPin, HIGH);       //turns off the laser  HIGH is off  (module JQ3CF- 05vDC works fine)
    Serial.println (numberForPython);   //goes out to python
    now_time = millis(); 
    delay (100);
    Serial.println (now_time); //this will also go out to Python;  could be useful
    
    old_time = (now_time + 30000);//  30000 would be 30 seconds.  this establishes the interval
  }

  delay (100);
  now_time = millis();  //this is the time stamp in milliseconds----  1000 per second
  
}

///////////////////////////////////


import serial #import serial library
import sqlite3
import datetime
import sys
from random import randint
python_script = sys.argv[0]
data_object=serial.Serial("COM7",baudrate=9600 , timeout=3.0)  #  data_object is the object!!
###################
conn = sqlite3.connect ('db_millis_oct_2023')
c = conn.cursor()


c.execute("CREATE TABLE IF NOT EXISTS millis_table (column_one TEXT,column_two REAL, python_script )")
#####    

    
   
##conn.commit()
    ###

    ##########################
print ("delay 2")


while (1==1):
    (data_object.inWaiting()>0)  

    unix = (datetime.datetime.now())
       
    
                         ##  this is the input from Arduino
    ##unix = (datetime.datetime.now())
    ##    make the unicode into string then into int
    
    #rand_interval =(randint(17,21))

    in_data = data_object.readline().decode('ascii')   ##
    in_data = in_data.encode('utf-8')  #  ??  THAT DID IT!!!!
    print (in_data)
  #  if (in_data != NONE):
  
        
        ##in_data = strip(in_data)
    c.execute("INSERT INTO millis_table(column_one,column_two,python_script) VALUES (?,?,?)",
		 ( unix , in_data ,python_script ) )
    conn.commit()

####end###########################

///////////////////////////////

I have moved to a different approach and still have the same problem. This is using the millis() function. Compare time now to time plus increment. If true then do it, otherwise wait. I had a bit of trouble with it. I am more careful with the code. (My development computer is not my regular computer)
The sqlite3 is way easier than mysql. The Python requires the sqlite3 library. I assume that this requires the sqlite3 software. The Python puts the database in the same folder, at least the way I have it configured. With the Arduino running, kill the serial monitor before starting the Python. I do not remember this issue with other projects. If you download this code and can run it I would be amazed. Advanced beginners, like me, find the system part of the project terrifying.

Is there an if missing there? Or a while? It's odd to just have a boolean conditional in parenthesis by itself like that.

Your method will work for a short time, but is not stable long term. At around 49 days it will fail. You should always keep track of the last time and calculate the interval to now instead of trying to predict a future time. The method employed in the classic "Blink Without Delay" example will safely run and avoid this bug.

This is am is an important observation: The Arduino Serial Monitor does not show the line breaks. But the Python monitor shows this extra junk. There are exactly the same ten of them between the data. This implies that the issue is with the Python.
I mostly do not understand the code.

@mrphysh try receiving the numbers in the Python script first, if you can do that successfully then move on to adding the data to the database.

In the Arduino only send "numberForPython" and see if the following script will print it. Let Python add the time stamp for now.

This script is very close to what you already have and it prints an ascii value as an integer with a time stamp

import serial 
import sqlite3
import datetime
import sys
from random import randint

python_script = sys.argv[0]
serial_object=serial.Serial("COM7",baudrate=9600 , timeout=0.5)  

def read_com_port():

	while True:
		current_datetime = datetime.datetime.now()
		formatted_time = current_datetime.strftime('%m/%d/%Y  %H:%M:%S')

		if serial_object.in_waiting > 0: #if we have data in the serial buffer print it or else do nothing
			try:
				in_data = serial_object.readline().decode('ascii')
				in_data = in_data.encode('utf-8')
				print (int(in_data),"   ",formatted_time)
			except:
				pass

read_com_port()
1 Like

very cool. I hope and suspect that this could be useful for a different project.
This is documented at GitHub

[mrphysh/magnetic-north: Is magnetic north constant? (github.com)]

(GitHub - mrphysh/magnetic-north: Is magnetic north constant?)

Arduino Code

//This is the delay platform
//my delay functions are not working correctly.  I am just now understanding it!!
//  this is also a practice platform  for the laser 4.5 volt relay

const int relayPin = 8;  //output...  at intervals, the laser is turned on

long  numberForPython = 8888;
long now_time = 340000 ; //  just pick a value
long old_time  =  2000;  //  must be lower than 340000

///////////////////////////////////////
void setup()
{

  pinMode (relayPin, OUTPUT);
  digitalWrite (relayPin, HIGH); //  relay HIGH is the default off;  'normally open' is default according to how I have it wired

  Serial.begin(9600);
  Serial.println("now time and old time.  30000 delay");  //this is a 2 minute daley  and my wake up for the python
  numberForPython = 887777;
  Serial.println (now_time);
  Serial.println (old_time);

}
/////////////////////////////////////////
void loop()    {
  //Serial.println (numberForPython);        //this goes out to Python and  this is my  "start Python" signa
   //  this goes into the database as a series of numbers
                                           //  This gives us the number of iterations
  //now_time = millis();
  delay(400);
  //Serial.println (now_time);
  if (now_time > old_time) {

 numberForPython = (numberForPython + 1);   //  this should be nice and sequential
   
    digitalWrite (relayPin, LOW);    //  low is on ..  green light on   turn on the laser
    //the green light should come on  (module JQ3CF- 05vDC works fine)
    // for a few seconds LASER ON
 /*  this is the locaion of the 40 LDR sensors.  I have it as 40 'if' statements.
         only one should go positive and that is sent out to Python and put in a
         database.
        */
    delay(1000);
    digitalWrite(relayPin, HIGH);       //turns off the laser  HIGH is off  (module JQ3CF- 05vDC works fine)
    Serial.println (numberForPython);   //goes out to python
    now_time = millis(); 
    delay (100);
    //Serial.println (now_time); //this will also go out to Python;  could be useful
    
    old_time = (now_time + 30000);//  30000 would be 30 seconds.  this establishes the interval
  }

  delay (100);
  now_time = millis();  //this is the time stamp in milliseconds----  1000 per second
  
}

////////////////////////////////////////////////////////////////
Python code

import serial #import serial library
import sqlite3
import datetime
import sys
from random import randint
python_script = sys.argv[0]
data_object=serial.Serial("COM7",baudrate=9600 , timeout=3.0)  #  data_object is the object!!
###################
conn = sqlite3.connect ('db_millis_oct_2023')
c = conn.cursor()


c.execute("CREATE TABLE IF NOT EXISTS millis_table (column_one TEXT,column_two REAL, python_script )")
#####    

    
   
##conn.commit()
    ###

    ##########################
print ("delay 2")


while (1==1):
    unix = (datetime.datetime.now())

    if     (data_object.inWaiting()>0):
        try:
            in_data = data_object.readline().decode('ascii')
            in_data = in_data.encode('utf-8')
            print (in_data)
            c.execute("INSERT INTO millis_table(column_one,column_two,python_script) VALUES (?,?,?)",
		 ( unix , in_data ,python_script ) )
            conn.commit()
        except:
            pass

####end###########################
I like your input regarding the format of the UNIX time stamp. But I am getting used to their default and afraid to mess with it.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.