simple averaging troubles

I have got my GPS clock running fine, it generally runs on a 60 second " blink without delay" sketch relying on the 16 MHz xtal, but every time there is a new minute from the GPS module, it synchronises with that.

Thats working fine, but then I tried counting the milliseconds between minutes from the GPS, which would be 60000 if the sketch and xtal were perfect.

I then used this count, which was always within 10 mS of the 60000, as the “interval” count for the Arduino timer

If I lost GPS lock, it would carry on using the corrected interval for the Arduino timer.

The “error” drifts around a bit, so I want to average out the last 10 readings, and threw together a simple test sketch ( below )

I assumed that by assigning a value to an array cell, it would overide the previous data there, but it seems to add to it?. the total10times just gets up to the millions.

There are some averaging sketches I found, but more complicated than I thought I need - it looked a simple task - what have I missed ?

(The last line is to ignore the result if its out of range.)

       times [x] = since;   // times is the array holding all the millis "since" the previous minute
                  x++;                     //  index increment 0 -  9
                  if ( x==10 ){ x = 0;}
                        Serial.print("x =  "); Serial.println(x);        
                  unsigned long total10times = times [0] + times [1] + times [2] + times [3] + times [4] + times [5] + times [6]
                   + times [7] + times [8] + times [9];
                    Serial.print("total10times == "); Serial.println(total10times);
                   
                  unsigned long average = total10times / 10;
                    Serial.print("average == "); Serial.println(average);
             if ( (average < 61000 ) && ( average > 59000)) { interval = average ; }  else { interval = 60000 ; }

Does each times [ x ] = a ( currentMillis - previousMillis) type number which is around 60,000 (1 minute ) ?

Yes Bob, I defined it as unsigned long times [10];

I have since reduced the samples to 3, but the total is still all over the place

How did you calculate:

times [x] = since;

And have you tried printing out each value of times[] and not just the total?

Are you only calculating total10times when there are known values in times[]? e.g. does total10times get calculated even before you have filled 10 new samples into the times[] array, which may or may not have known values?

This is a good example of why posting just "this is the code I'm having problems with" doesn't usually lead to an answer.

Thanks James, you are spot on - both counts :slight_smile:

If I had posted all of the sketch, you ( and 50,000 other Arduino heads ) would immediately have seen that when I had captured the time of receiving the minute change from the GPS module, I had called it "int present = millis () " instead of unsigned long !

( giving me 43 million as "since" .)

Its working well now, except I must tell it to ignore any " since" over 61000 so that it only averages counts between subsequent minutes.

( I didn't post the whole sketch as it was 75% commented out lines :disappointed_relieved: )

I have just spotted a simple running averager by macegr on adafruits forum I might play with, it looks like it might be quite smooth :-

float adcResult = 0;
void setup()
{  Serial.begin(9600); }

void loop()
{
  adcResult = adcResult * 0.95 + analogRead(0) * 0.05;
  Serial.println(adcResult);
  delay(50);
}

That averaging works very well, I have incorporated a MacGuyvered rounding up or down the decimal places to keep the interval of the freerunning timer within 1 millisecond , which ( ideally , if the temperature of the xtal didnt change too much )would equate to 43 seconds error a month !
Hopefully the GPS will not drop out for anything like that !

But the reading from the GPS between minute changes varies quite a few milliseconds, I suppose it depends on where the loop my sketch was when the next data packet is ready.
An interrupt would sort that out, but doesn’t it mess up the millis count?

I am not really concerned as any delay in reading the minute change would be cancelled out the next minute anyway.

I am using the 5% averaging swaps, which I think relates to averaging over 20 samples ( although the history is never erased, just slowly modified ) but as I am smoothing a very steady level, I could take it down to 1% perhaps , I have days to let it settle.

it looks like my freerunning clock would have been 6 milliseconds fast every minute without this calibration. If the GPS drops out it will carry on with the corrected interval.

Heres the code and below it the readings as it averages with 5% swaps

/*
   GPS clock by John Smith aka Boffin1  based on  http://tronixstuff.com/tutorials > Chapter 19
 and code by Aaron Weiss of SparkFun Electronics;  http://bit.ly/99YnI6
 and code and libaries by arduiniana.org. and the patirnt people of the Arduino forum !
 Thank you :)
 */

#include <NewSoftSerial.h>
#include <TinyGPS.h>


long error;
long clockmillis = 0;
long rxmillis = 0;
long interval = 60000;
int x =0;
unsigned long since;
unsigned long times [10];
const byte digitTable [10] = {                     //   patterns for 7 segment displays
  B11111110, B10110000, B11101101, B11111001, B10110011,  //  first bit for switching on display
  B11011011, B11011111, B11110000, B11111111, B11111011};

float average = 60000;
int clockmins;
int clockhours;

#define RXPIN 14 //   
#define TXPIN 15 // 
#define GPSBAUD 9600 // baud rate of our UP501D GPS module. Change for your GPS module if different

// Create an instance of the TinyGPS object
TinyGPS gps;
// Initialize the NewSoftSerial library to the pins you defined above
NewSoftSerial uart_gps(RXPIN, TXPIN);

// This is where you declare prototypes for the functions that will be 
// using the TinyGPS library.
void getgps(TinyGPS &gps);

int hourtens;
int hourunits;
int mintens;
int minunits;
int rxhours;
int rxmins;

void setup()
{
  uart_gps.begin(GPSBAUD); // setup sketch for data output speed of GPS module
  Serial.begin(115200);
  interval = 60000;
}
//  ========================================================================


void getgps(TinyGPS &gps)    // The getgps function will get the values we want.
{
  int year,a,t;
  byte month, day, hour, minute, second, hundredths;
  gps.crack_datetime(&year,&month,&day,&hour,&minute,&second,&hundredths);
  hour=hour+2; // my zone is GMT +2
  if (hour>23) { 
    hour=hour-24;
  }

  rxhours = ( int ) hour;
  rxmins = (int) minute;

  if ( rxmins != clockmins ) {                    // ===================MINUTES HAVE CHANGED THIS SECOND ==================
    unsigned long  present = millis ();          //  set actual millis time of minute change for all subsequent calcs and timings
    since = (present - rxmillis) ;              // the millis count since the last minute change ( GPS accuracy on average )
    rxmillis = present;                        //  millis count at update
    Serial.print("rxmin has just changed,");  //  update clock, at secs = zero
    clockmins = rxmins;  
    clockhours = rxhours;     
    clockmillis = present ;                    // reset freerun clock counter millis
    Serial.print("  time = ");   
    Serial.print(clockhours); 
    Serial.print(":");   
    Serial.print(clockmins);
    if ( (since > 59000 ) && ( since < 61000 )) {  //  only calibrate the freerun timer if consecutive GPS minutes
      Serial.print(" since = ");  Serial.print(since);
      average = average * 0.95 + since * 0.05;  // deduct 5% of the average, and add 5% of the new reading
      Serial.print("  averaged since = ");   
      Serial.print(average);
      if ( (average < 61000 ) && ( average > 59000)) //  if its already locked on
      
      { float error = (average - ( long ) average);
        Serial.print(" error = "); Serial.println(error);
        if ( error >= .5 ) { average ++ ; }
        if (error <= -.5 ) { average --;  }
               interval = (long ) average ; 
      } //  end of if locked on
      Serial.print(" interval rounded for free running timer = "); Serial.println(interval);
    }   

    showtime ();        
    // =====================================================================================================        
  }

}

//********************************************************************************************

void loop()
{
  while(uart_gps.available())     // While there is data on the RX pin...
  {
    int c = uart_gps.read();    // load the data into a variable...

    if(gps.encode(c))      // if there is a new valid sentence...
    {
      getgps(gps);         // then grab the data, 
    }
  }

  //Serial.println("running clock    ");
  unsigned long currentMillis = millis();
  if(currentMillis - clockmillis > interval) {
    clockmillis = currentMillis;
    clockmins ++; 
    if ( clockmins >59 ) { 
      clockmins = 0; 
      clockhours ++; 
      if (clockhours >59 ) { 
        clockhours = 0 ;
      }
    }// end of if mins>59 
    Serial.print("  interval= "); 
    Serial.print(interval);
    Serial.print("clock time   == "); 
    Serial.print(clockhours); 
    Serial.println(clockmins); 
    showtime ();

  }//  end of if current - clock > interval

  // Serial.print("clock time   == "); Serial.print(clockhours); Serial.println(clockmins); 
}  // end of loop


and heres some of the data being averaged , 


 interval rounded for free running timer = 60000
rxmin has just changed,  time = 12:49 since = 60008  averaged since = 60000.54 error = 0.54
 interval rounded for free running timer = 60001
rxmin has just changed,  time = 12:50 since = 59980  averaged since = 60000.46 error = 0.46
 interval rounded for free running timer = 60000
rxmin has just changed,  time = 12:51 since = 60013  averaged since = 60001.08 error = 0.09
 interval rounded for free running timer = 60001
rxmin has just changed,  time = 12:52 since = 60013  averaged since = 60001.68 error = 0.68
 interval rounded for free running timer = 60002
rxmin has just changed,  time = 12:53 since = 60008  averaged since = 60002.94 error = 0.95
 interval rounded for free running timer = 60003
rxmin has just changed,  time = 12:54 since = 59980  averaged since = 60002.75 error = 0.75
 interval rounded for free running timer = 60003
rxmin has just changed,  time = 12:55 since = 59995  averaged since = 60003.31 error = 0.31
 interval rounded for free running timer = 60003
rxmin has just changed,  time = 12:56 since = 60011  averaged since = 60003.69 error = 0.69
 interval rounded for free running timer = 60004
rxmin has just changed,  time = 12:57 since = 59984  averaged since = 60003.66 error = 0.66
 interval rounded for free running timer = 60004
rxmin has just changed,  time = 12:58 since = 60014  averaged since = 60005.12 error = 0.12
 interval rounded for free running timer = 60005
rxmin has just changed,  time = 12:59 since = 60011  averaged since = 60005.41 error = 0.41
 interval rounded for free running timer = 60005
rxmin has just changed,  time = 13:0 since = 59995  averaged since = 60004.89 error = 0.89
 interval rounded for free running timer = 60005
rxmin has just changed,  time = 13:1 since = 60008  averaged since = 60006.00 error = 1.00
 interval rounded for free running timer = 60006
rxmin has just changed,  time = 13:2 since = 59991  averaged since = 60006.19 error = 0.20
 interval rounded for free running timer = 60006
rxmin has just changed,  time = 13:3 since = 60012  averaged since = 60006.49 error = 0.49
 interval rounded for free running timer = 60006
rxmin has just changed,  time = 13:4 since = 59981  averaged since = 60005.21 error = 0.21
 interval rounded for free running timer = 60005
rxmin has just changed,  time = 13:5 since = 60006  averaged since = 60005.25 error = 0.25
 interval rounded for free running timer = 60005
rxmin has just changed,  time = 13:6 since = 59992  averaged since = 60004.59 error = 0.59
 interval rounded for free running timer = 60005
rxmin has just changed,  time = 13:7 since = 60006  averaged since = 60005.61 error = 0.61
 interval rounded for free running timer = 60006
rxmin has just changed,  time = 13:8 since = 60000  averaged since = 60006.28 error = 0.28
 interval rounded for free running timer = 60006
rxmin has just changed,  time = 13:9 since = 59998  averaged since = 60005.86 error = 0.86
 interval rounded for free running timer = 60006

You might want to check but if Serial read and print use interrupts then the millis() clock will be stopped during those. If so then you might find a way to account for that and include it in your MacGuyvering.

BTW, if you never grow up then you'll be a better programmer.

Thanks GoForSmoke - I completely forgot to comment out all the zillion SerialPrints again !
I will try it with just printing the interval and see how it looks .

I will never get old , only in numbers , kid at heart :slight_smile:

Hmm The interval ends up at 60009 which doesnt make sense, so when I finally managed to get the GPS module to lose the signal ( by turning it upside down with the antenna pointing down, and the whole thing wrapped in foil, with an aluminium plate covering the lot ! ( see Arduino Forum ) the free running timing was way too slow.

By ignoring the calibration part of the sketch, and presetting the interval to 59851 it ran through the night with only a few seconds error.

I think the problem is to do with the variation of the timing I measure from the GPS minute changes.

As the GPS signal is so reliable, I am going to ditch the idea for now of calibrating the millis to the GPS, and let the GPS reset the time and clockmillis each minute.

When I get a chance I might revisit and try counting millis between hour changes from the GPS, which will reduce any latency problems I am getting n recording the minute change time.