How to handle sampling data when posting to web

Hi guys,

This is a concept question, not so much a specific question yet. I have fiddled around with both a current sensor (SCT-013) and a water flow sensor (Liquid Flow Meter - Brass 1/2 Nominal Threaded : ID 833 : $24.95 : Adafruit Industries, Unique & fun DIY electronics and kits). Meaning I have successfully set up an arduino to read their data and verified that it was “correct”. (within error)

Now Im ready to have that data read periodically (maybe 1x every hour) and sent to a db or something.

In the current code I use this:

#include "EmonLib.h"
EnergyMonitor emon1;
char ssid[] = "myssid";       // your network SSID (name)
char password[] = "mypwd";  // your network key

void setup(){
  Serial.begin(9600);
  emon1.current(1, 111.1);             // Current: input pin, calibration.111.1
}

void loop(){
double Irms = emon1.calcIrms(1480);  // Calculate Irms only
Serial.print(Irms*120.0);           // Apparent power - from 230 to 120
  Serial.print(" ");
  Serial.println(Irms);             // Irms
}

So the value is stored in a local variable. Same as in my water meter code. In both cases the variables are written to the serial constantly. I post the data every 60 minutes, using millis like this:

#include "EmonLib.h"
EnergyMonitor emon1;
char ssid[] = "myssid";       // your network SSID (name)
char password[] = "mypwd";  // your network key

unsigned long previousMillis = 0;        // will store last time was updated
const long interval = 180000; 
char irmsString[10];


void setup(){
  Serial.begin(9600);
  emon1.current(1, 111.1);             // Current: input pin, calibration.111.1
}

void loop(){
  //THIS SECTION MEASURES CURRENT CONSTANTLY AND PRINTS TO SERIAL MONITOR FOR DEBUGGING
  double Irms = emon1.calcIrms(1480);  // Calculate Irms only
  Serial.print(Irms*120.0);           // Apparent power - from 230 to 120
  Serial.print(" ");
  Serial.println(Irms);             // Irms
  
  //THIS SECTION TESTS FOR TIMESTAMP IN ORDER TO POST DATA OR NOT
  unsigned long currentMillis = millis();
  if (currentMillis - previousMillis <= interval) { //when time elapsed is LESS than interval
  } else { //if deltaTimeElapsed is GREATER than target interval, sample and post...
    previousMillis = currentMillis; //sets previous-0 to current-3600 
    Serial.print("connecting to ");
    Serial.println(host);
  
    // Use WiFiClient class to create TCP connections
    WiFiClient client;
    const int httpPort = 80;
    if (!client.connect(host, httpPort)) {
      Serial.println("connection failed");
      return;
    }
  
    // We now create a URI for the request
    dtostr(Irms, 4, 2, irmsString);
    String url = String("/emoncms/input/post.json?apikey=mykey&node=fortnite&json={\"i\":") + irmsString + "}";
    Serial.print("Requesting URL: ");
    Serial.println(url);
  
    // This will send the request to the server
    client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" + 
               "Connection: close\r\n\r\n");
    delay(500);
  
    // Read all the lines of the reply from server and print them to Serial
    while(client.available()){
      String line = client.readStringUntil('\r');
      Serial.print(line);
    }
  
    Serial.println();
    Serial.println("closing connection");
    }
}

The problem is that if power goes out, the Arduino restarts but the data starts from zero again. So if I wanted to log water or power consumed the final data would be incorrect if power ever went out. How do I solve this issue?

Any suggestions on how to accomplish this?

I can't really see what you're doing but I assume that the Arduino is accumulating a total of water or power consumed and posting it.

Why not just accumulate how much you've used since the last time you sent it to the server? Then if the power goes out, you've only lost your recent measurements.

Let the server add it up for you to get longer term usage.

Ok i could send it to the server or not necessarily. I just thought of the server as a way to ensure persistence. Is there a better way? Because if the power goes out so does the internet, so if it is being sent every 1 hour and power goes out 59 minutes after the last post to the server, i might loose considerable amounts of data. So if there is a better way Id appreciate the suggestion.

As for posting to the web for persistence, so I would add a lastPostTimestamp and set it to the local time of the last posting. So the webPosting() function - so to speak - would include setting that time and then when the arduino rebooted it would fetch "the last data value posted" from the web and use it to accumulate new data over that?

Save it to an SD card perhaps.

Whether server or SD, just save it frequently enough for your needs - perhaps you don't care if you lose ten minutes data. Perhaps it's a second's worth that matters to you.

Maybe get a battery to power your Arduino during an outage. Or some caps that keep it up long enough to save the current readings to EEPROM if you're not using an SD.

Marciokoko:
if it is being sent every 1 hour and power goes out 59 minutes after the last post to the server, i might loose considerable amounts of data. So if there is a better way Id appreciate the suggestion.

You can post to the server more frequently or ...

Marciokoko:
As for posting to the web for persistence, so I would add a lastPostTimestamp and set it to the local time of the last posting. So the webPosting() function - so to speak - would include setting that time and then when the arduino rebooted it would fetch "the last data value posted" from the web and use it to accumulate new data over that?

If you wanted to do infrequent dumps to the server you can write to some nonvolatile memory on your microcontroller as wildbill was suggesting. If you then want to post to your server every hour, day, year, when you are running low on nonvolatile storage or when the Internet is available (any condition you like) you can read from the nonvolatile memory and post to the server. You can do that with timestamps or an incremented id. Then compare what the server has and delete entries in your nonvolatile memory so that you free up space.

I don't know what microcontroller you are using, but I use the ESP32 and ESP8266. It can write files to flash using SPIFFS. It works just like any file system and writes to nonvolatile flash memory. You can write a considerable amount of data there and then post the data to a server later.

Well I think in the end I'll have Arduino nanos reading the data from the sensors. So I could write data to the server via an Esp8266 module. I'm thinking of using a public instance of emoncms. Here is the thing, if power goes out the internet usually goes down as well and that's at least 1x a month for 8-12 hours.

In this case I would need a battery backup for the arduino/esp/sensor. But I'd have to store the data locally while I have battery but no internet. Once I got the internet back I'd have to figure out a way to update the data.

So for example, for power readings I'd have maybe 500 wh accumulated at the moment the power went down. At this point have to log to the SD card starting at 01 again. Then when the power came back I'd have to figure out a way to take the SD log value, say 98 wh, and add them to the previous value of 500 for a new running total of 598 wh and continue.

But I'd have to do some fancy dB reading first in order to fetch the last value.