Solar logger lost data

Using Arduino Programming Questions
1

I have an ESP32 that gets pulses from my electricity meter. The pulses are aggregated over a day and the system provides a readout of values on a web page, and saves daily totals in a SPIFFS file named yyyy_mm.csv.
Its been working fine but on 11 May it lost its data, as shown here. dTotalkWh and mWatts are both incorrect - but NOT zero as they would be if zeroStats had been called.
The device is powered from a USB wallwart but (as far as I know) we have not had any power outages.

I've attached the rest of the sketch.
datavalues.ino (3.1 KB)
SolarLogger.h (6.9 KB)
SolarLoggerFinal2.ino (5.4 KB)
spiffs.ino (704 Bytes)
webpage.ino (1.1 KB)
wifi.ino (875 Bytes)

Day,total kWh,maxWatts

1, 15.292, 2990
2, 8.395, 3070
3, 8.555, 2050
4, 5.414, 1500
5, 19.441, 3180
6, 18.462, 3080
7, 14.466, 3200
8, 3.193, 970
9, 14.187, 3160
10, 13.143, 3190
11, 0.031, 50
12, 10.214, 3160

I've pored over the sketch but I cant see how that can happen. Here is the code (datavalues.ino) that does the calculation.

//preparation and calculations for data

// detect and count up pulses
void IRAM_ATTR ISR() {
  // The IRAM_ATTR attribute places compiled code in the (faster) Internal RAM (IRAM) of the ESP32.
  tNow = millis();
  //max pulse rate is 1 per sec; 3600 watts in 3600 sec = 3.6kWh
  if (tNow - tLast > 300) { // prevent bounce
    pCount++; //increment count
    cWatts = pCount * 10; //cWatts, pWatts, mWatts are adjusted (*10) as count is over 6 min not 60min
    tLast = tNow;
  }
}

//calculate wattage and power values

void zeroStats() {  //zeroStats is called by setup and in newDay
  pCount = 0; //reset pulse count
  cWatts = 0; //reset current  watts
  mWatts = 0; //reset maximum watts
  pWatts = 0; //reset previous period
  dTotalkWh = 0; //reset total for current day
}

//every 6 minutes:
void checkPeriod() {
  if (timeinfo.tm_min != cMin) { // new minute
    cMin = timeinfo.tm_min;
    if ((cMin % 6) == 0) { // end of previous period
      pWatts = cWatts;
      if (cWatts > mWatts) {
        mWatts = cWatts; // update maximum value
      }
      dTotalkWh += pCount / 1000.0;  //convert watts to kW & add to day total
      Serial.println(&timeinfo, "%H:%M:%S");  //eg 07:04:45
      Serial.print("cWatts: ");
      Serial.print(cWatts);
      Serial.print("; mWatts:  ");
      Serial.print(mWatts);
      pCount = 0; //reset pulse counter
    }
  }
}

void newDay() {
  //format & save 1 line of data
  snprintf(dataString, sizeof(dataString), "\n%3u, %6.3f, %3u", currDay, dTotalkWh, mWatts);
  appendFile(SPIFFS, fileName, dataString);
  Serial.print("newDay: ");
  Serial.println(dataString);
  yTotalkWh = dTotalkWh; //store current total as "yesterdays total"
  zeroStats();
}

void newMonth() {
  //create new filename for saving data and append header line; handle new day first; filename needs leading /
  strftime (fileName, sizeof(fileName), "/%Y_%m.csv", &timeinfo); // "/" + 4 digit year + "_" + 2 digit month +".csv" + null = 12 chars

  if (SPIFFS.exists(fileName)) {}
  else { //no file ready
    writeFile(SPIFFS, fileName, headerString); //char headerString[30] = "Day,total kWh,maxWatts \n";
    Serial.print("Filename is ");
    Serial.println(fileName);
  }
  currMonth = timeinfo.tm_mon; //update current month
  Serial.print("newMonth: ");
  Serial.println(headerString);
}

grateful for any help.

2

John, we had quite energetic thunderstorms on 11th here in West Yorkshire, did you get similar? I'm wondering if a momentary power brown-out could have caused the esp to reset or something?

3

I'd consider writing the data more often to a file so that it is safe from power failures etc., then have a daily job which runs to generate the totals for that day.

4

Thanks Paul, we did. I considered adding a big electrolytic cap across the power rails, but with the ESP drawing around 60mA it wouldnt supply it for very long.
Looking at my network log file I see there were some lost pings around 7/8 pm which if there was a reset then would explain the very small but non-zero value.

Thanks @6v6GT (Is that a beam tetrode with go faster stripes?)
A much better solution, Ill need to think how I implement it. Perhaps I could:
new day: create a "today" SPIFFS file
save the 6min totals as they come in;
new day:
read and sum the 6 min totals to generate the day total and (less important) the max value
open and save those results to the "month" file.
delete the "today" file & re-create it so its empty.

No idea how to write and read integer values to SPIFFS, the example is exactly NO help. I guess its the same as reading and writing to serial, so I'll look it up. More work, but it keeps the old brain alive.

5

GT : glass envelope, small.

6

Anyone help how I write and read back integers in SPIFFS?

7

Well you can start here by attempting to read the file. ESP32 Arduino SPIFFS: Reading a file - techtutorialsx
It does it one character at a time. You have to assemble each line in a buffer and parse it.
It looks like you are writing lines in this format into the file at the moment:

snprintf(dataString, sizeof(dataString), "\n%3u, %6.3f, %3u", currDay, dTotalkWh, mWatts);

I guess your temporary file would need only dTotalkWh

1 Like
8

This may not be of interest as it would be a big change, but whenever I see ppl strukkling with SD card libraries and SPIFFS or one of the other file systems, I feel compelled to argue for

openlog open source data logging.

You can fix up your program to print exactly the data you want saved, then just route that to an openlog device on a serial pin.

It let me skip over all kindsa pain.

I’ve used cheaper versions of the device. I believe any time it failed could be traced to something I did or didn’t do; if I was flying it to Mars I’d prolly go ahead and spend the < $20 on the real deal.

Pop the chip out and when you look at it on a real computer, it might could already be formatted for human consumption.

a7

9

Thanks @6v6gt
snprintf int to buffer& file.println(buffer)

reading back

char buffer[10]; //space for single integer
while (file.available()) {
 int k = file.readBytesUntil('\n', buffer, sizeof(buffer) -1);
 buffer[k] = 0;  //readbytesUntil - The terminator character is discarded from the stream.
 Serial.println(buffer);
}

great link and good onward links

10

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.