Seeking assistance identifying internal damage

For the past 2 years I've been putting together a hydroponics controller and adding bits of code as I learn them. If not for the constant failures, I would be extremely proud with the way the project works (when it's actually working), but it seems the RTC can only maintain time for at most 3 days, but it usually fails sooner. Perhaps this isn't even an RTC issue, I don't know how to troubleshoot this to determine exactly what is wrong and how to address that.

My circuit is a Freetronics Ethermega. A DS3231 Chronodot from Adafruit is the RTC, but I'm using the DS1307 library due to difficulties with other libraries. Also connected are 2 DHT22 temp/humidity sensors. There is also a Sainsmart 8 channel relay that is isolated completely from the Arduino as no grounds are shared. Instead, a 5V master lead supplies the optoisolators and the digital pins sink when instructed to do so. Finally, I also have an 8 head peristaltic pump dosing module. Each head is controlled by a TIP120 circuit with flywheel diode and resistor between gate and digital pin.

The networking aspect of my project is completely handled by Blynk. With Blynk, I can control certain things from my smartphone with their free app and really bring out the IoT from Arduino. This is how I can choose between pumps and dosing amounts. I also use the same approach for dosing out reverse osmosis water in 1/4 gallon increments. I also have it set up so all Serial.print type strings are printed to a monitor in the Blynk app with each revolution of a loop.

I don't know exactly how long the EM will run till it fails, but at most is roughly 3 days, all the way down to just a few minutes. When the failure occurs, the relay states remain in the programmed position at the time of failure and the RTC time would simply freeze. However, recently, the RTC began to display weirdly as "153:165:85-PM 165/165/2165". Some times is would display the current time minus a few hours and minutes, day and month as 00, but the correct year. At each failure, I reset power to the MCU and begin watching to see how long the connection remains and what the time will look like when it begins sending again. My first thought was a faulty RTC, so I ordered another from Adafruit and still the same result. During the construction, I may have incorrectly wired some stuff and connected everything to discover nothing happening. Perhaps this damaged the MCU somehow by sending 5 volts and a GND of a device and vice-a-versa? I am of the mind to replace the Ethermega just to rule it out, but another $120 is a steep "what if".

I apologize if this is too wordy, or is lacking in other details. If more information is needed, please ask and I will provide. Here is my sketch.

#include <SPI.h>                    // Used by Blynk
#include <Ethernet.h>               // Used by Blynk
#include <BlynkSimpleEthernet.h>    // Used by Blynk
#include <Wire.h>                   // I2c
#include "DHT.h"                    // DHT data wire connected to I/O pin with a 10k pullup resistor to 5V               
#include <SimpleTimer.h>            // used to run functions at preset intervals
#include "RTClib.h"                 // RealTimeClock Library for DS1307 and DS3231
#define BLYNK_PRINT Serial
//  DHT Reference Values - CHANGE THESE TO MANAGE YOUR ROOM'S CLIMATE - //
byte hiMaxTemp = 80;   // temp that triggers heat removal device(s) on
byte lowMaxTemp = 70;  // temp that triggers heat removal device(s) off
byte hiMinTemp = 70;   // temp that triggers heater on
byte lowMinTemp = 60;  // temp that triggers heater off
byte hiHum = 55;       // High humidity value that triggers dehumidifier on
byte lowHum = 50;      // Low humidity value that triggers dehumidifier off

//-Digital Pins - Peristaltic Pump Variables - 00 Series Pins
int pumpPin[8] = { 2, 3, 4, 5, 6, 7, 8, 9 };
uint32_t multiplier[8] = { 858, 827, 872, 865, 887, 895, 913, 843 }; //ms per ml
uint32_t startPump = 0;
uint32_t runCount;
bool pumpRunning = false;
const char *nuteType[8] = { "GH Armor Si", "GH Flora Blend", "GH CALiMAGic", "GH Kool Bloom",
                            "GH Flora Gro", "GH Flora Micro", "GH Flora Bloom", "GH pH Down"
                          }; //Text Printed to Terminal Widget^^

float DOSEml;      //Step Widget (0.25 per step, send step/NO, loop values ON)
int button = 0;   //Button Widget set to Switch
int x;           //Correlates Array Positions with Pump Motor Pins
BLYNK_WRITE(V4) {
  x = param.asInt() - 1;
}
BLYNK_WRITE(V5) {
  DOSEml = param.asFloat();
}
BLYNK_WRITE(V6) {
  button = param.asInt();
}
//
uint32_t msPerGallon = 34000; //ms per gallon
uint32_t ROstart = 0;
uint32_t countGallons;
bool runningRO = false;
int pumpAon;          //Blynk Override to turn pumpA back on
int pumpBon;          //Blynk Override to turn pumpB back on
int ROpumpOn = 0;     //Blynk Triggered RO Pump
float totalGallons;   //Number of RO Gallons Selected in Widget
BLYNK_WRITE(V9) {
  pumpAon = param.asInt();  // pumpA remote
}
BLYNK_WRITE(V10) {
  pumpBon = param.asInt();  // pumpB remote
}
BLYNK_WRITE(V11) {
  ROpumpOn = param.asInt();  // ROpump remote
}
BLYNK_WRITE(V12) {
  totalGallons = param.asFloat();  // ROpump remote
}

// Digital Pin 10 is <RESERVED> for W5100 Wiznet chip
// Digital Pins - 8 Channel Relay Assignments - 20 Series Pins - 120VAC~ switching
#define lightA 22             //Relay 1/a  
#define lightB 23             //Relay 2/b  
#define pumpA 24              //Relay 3/c 
#define pumpB 25              //Relay 4/d  
#define ROpump 26             //Relay 5/e  
#define scrubberFan 27        //Relay 6/f  
#define heatVentA 28          //Relay 7/g  
#define heatVentB 29          //Relay 8/h 

//-Digital Pins - 8 Channel Relay Assignments - 30 Series Pins - 120VAC~ switching
// -Dpins 30-37 are reserved for another bank of Relays.

//Digital Pin Valves Assignments - 40 Series Pins - Hi/Lo Solenoid Valves
//const byte valve[13] = { 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52 };

#define TURN_ON 0 // TURN_ON and TURN_OFF are defined to account for Active LOW relays
#define TURN_OFF 1 // Used to switch relay states for on/off of 120VAC~ devices  

RTC_DS1307 RTC;
float UTCOffset = -5.0;    // Your timezone relative to UTC (http://en.wikipedia.org/wiki/UTC_offset)
char auth[] = "PasteAuthKeyBetweenQuotes";
WidgetLCD lcdA(V7);  //Set LCD widget to Advanced Mode - Widget to display project clock
WidgetTerminal terminal(V8);
DHT dhtA(A0, DHT22);     // DHT instance named dhtA, (I/O pin, sensor type)
DHT dhtB(A2, DHT22);     // DHT instance named dhtB, (I/O pin, sensor type)
SimpleTimer timer;       // SimpleTimer instance named timer

void setup()
{
  Serial.begin(9600);
  //Serial2.begin() // For other baud rates
  //Serial3.begin() // For other baud rates
  Blynk.begin(auth);
  dhtA.begin();
  dhtB.begin();
  pinMode(A0, INPUT_PULLUP);    // DHT22 use internal 20k pullup resistors
  pinMode(A2, INPUT_PULLUP);
  Wire.begin();
  RTC.begin();
  //RTC.adjust(DateTime(__DATE__, __TIME__));
  //Uncomment "RTC.adjust" the first time this sketch is uploaded to program the RTC.
  //Once done, re-comment that line out, and upload this sketch again.

  for (int allRelays = lightA; allRelays <= heatVentB; allRelays++) {
    pinMode(allRelays, OUTPUT);
    digitalWrite(allRelays, TURN_OFF);
  }
  for (int p = 0; p <= 7; p++)
  {
    pinMode(pumpPin[p], OUTPUT);
  }
  Blynk.virtualWrite(V4, 0);
  Blynk.virtualWrite(V5, 0);
  Blynk.virtualWrite(V6, 0);
  Blynk.virtualWrite(V9, 0);
  Blynk.virtualWrite(V10, 0);
  Blynk.virtualWrite(V11, 0);
  Blynk.virtualWrite(V12, 0);
  while (Blynk.connect() == false) {}
  timer.setInterval(2000L, timeRoutine);      // 2 second intervals between timed routiness
  timer.setInterval(5001L, climateRoutine);   // 5 second intervals between climate routines
  timer.setInterval(207L, dosingPumps);       // 0.2 second interval to maintain accuracy (+/- .25)
  timer.setInterval(1003L, ROcheck);          // 1 second interval between RO pump routines
}

void loop()
{
  Blynk.run();
  timer.run();
}

Here's the 2nd half of the Sketch;

//Functions
void climateRoutine()
{
  byte h1 = dhtA.readHumidity();          // f1 and h1 are fahrenheit and humidity readings
  byte f1 = dhtA.readTemperature(true);   // from DHT/A
  byte h2 = dhtB.readHumidity();          // f2 and h2 are fahrenheit and humidity readings
  byte f2 = dhtB.readTemperature(true);   // from DHT/B
  Blynk.virtualWrite(V0, f1);      //  Set Virtual Pin 0 frequency to PUSH in Blynk app
  Blynk.virtualWrite(V1, h1);      //  Set Virtual Pin 1 frequency to PUSH in Blynk app
  Blynk.virtualWrite(V2, f2);      //  Set Virtual Pin 2 frequency to PUSH in Blynk app
  Blynk.virtualWrite(V3, h2);      //  Set Virtual Pin 3 frequency to PUSH in Blynk app
  //-------------------Bloom A Temp Test---------------------------------------//
  if (f1 >= hiMaxTemp)  //if "f1" is greater than or equal to hiMaxTemp,
  {
    digitalWrite(heatVentA, TURN_ON);  // TURN_ON heatVentA (fan).
    terminal.println("Exhausting the heat from Bloom A"); //  Text printed to terminal monitor
  }

  else if (f1 <= lowMaxTemp)  //  or else if "f1" is less than or equal to lowMaxTemp
  {
    digitalWrite(heatVentA, TURN_OFF); //  TURN_OFF relay E.
  }
  //-----------------------Bloom A Humidity Test-------------------------//
  if (h1 >= hiHum)  //if "h2" is greater than or equal to hiHum,
  {
    digitalWrite(heatVentB, TURN_ON);  // TURN_ON heatVentA (fan).
    terminal.println("Exhausting the RH from Bloom A"); //  Text printed to terminal monitor
  }
  else if (h1 <= lowHum)  //  or else if "h1" is less than or equal to lowHum
  {
    digitalWrite(heatVentB, TURN_OFF);
  }
  //-----------------------Bloom B Temp Test-----------------------------//
  if (f2 >= hiMaxTemp)  //if "f2" is greater than or equal to hiMaxTemp,
  {
    digitalWrite(heatVentB, TURN_ON);  // TURN_ON heatVentA (fan).
    terminal.println("Exhausting the heat from Bloom B"); //  Text printed to terminal monitor
  }
  else if (f2 <= lowMaxTemp)  //  or else if "f2" is less than or equal to lowMaxTemp
  {
    digitalWrite(heatVentB, TURN_OFF);
  }
  //-----------------------Bloom B Humidity Test-------------------------//
  if (h2 >= hiHum)  //if "h2" is greater than or equal to hiHum,
  {
    digitalWrite(heatVentB, TURN_ON);  // TURN_ON heatVentA (fan).
    terminal.println("Exhausting the RH from Bloom B"); //  Text printed to terminal monitor
  }
  else if (h2 <= lowHum)  //  or else if "h2" is less than or equal to lowHum
  {
    digitalWrite(heatVentB, TURN_OFF);
  }
  terminal.flush();
}

void timeRoutine()
{
  DateTime now = RTC.now();  // reads time at beginning of loop
  byte twelveHour = now.hour() - 12; // Variable used to display 13+ hours in 12 hour format
  byte zeroHour = 12;                // Variable use to convert "0" zero hour to display it as 12:00+
  byte displayHour;
  byte MIN = now.minute();
  byte SEC = now.second();
  char* meridian;

  if (now.hour() == 0)  // First we test if the hour reads "0"
  {
    displayHour = zeroHour;
    meridian = "AM";
  }
  else if (now.hour() >= 13)  // if no, Second we test if the hour reads "13 or more"
  {
    displayHour = twelveHour;
    meridian = "PM";
  }
  else
  {
    displayHour = now.hour();
    meridian = "AM";
  }
  char timeStamp[16];
  char dateStamp[16];
  sprintf(timeStamp, "%02d:%02d:%02d-%02s", displayHour, MIN, SEC, meridian);
  sprintf(dateStamp, "%02d/%02d/%04d", now.month(), now.day(), now.year());
  String ts;
  String ds;
  ts = timeStamp;
  ds = dateStamp;
  lcdA.clear();            //Advanced Mode
  lcdA.print(3, 0, ts);    //Advanced Mode
  lcdA.print(3, 1, ds);    //Advanced Mode
  //------------------12/12 BloomA Light - 5AM-5PM
  //Adjust hours and minutes in accordance with 24 hour time format.
  //Create tests where true is ON time and false is OFF time.
  boolean lightAstate = false;
  if (now.hour() >= 5 && now.hour() <= 16) lightAstate = true;
  if (lightAstate == true)
  {
    digitalWrite(lightA, TURN_ON);
    terminal.println("Lights On In Bloom A");  //  Text printed to terminal monitor
  }
  else
  {
    digitalWrite(lightA, TURN_OFF);
  }
  //--------------------12/12 Bloom B Light - 5PM-5AM
  //lightB is lit during the opposing 12 hours to lightA to conserve current draw from HID ballasts.
  boolean lightBstate = false;
  if (lightAstate == false) lightBstate = true;
  if (lightBstate == true)
  {
    digitalWrite(lightB, TURN_ON);
    terminal.println("Lights On In Bloom B");  //  Text printed to terminal monitor
  }
  else
  {
    digitalWrite(lightB, TURN_OFF);
  }
  //---------------Bloom A Feed Times------------------------
  boolean pumpAstate = false;
  if (pumpAon == 1) pumpAstate = true;
  if (now.hour() == 6 && now.minute() >= 0 && now.minute() < 5) pumpAstate = true;    //6:00 am - 5 mins
  if (now.hour() == 8 && now.minute() >= 30 && now.minute() < 35) pumpAstate = true;  //8:30 am - 5 mins
  if (now.hour() == 11 && now.minute() >= 0 && now.minute() < 5) pumpAstate = true;  //11:00 am - 5 mins
  if (now.hour() == 13 && now.minute() >= 30 && now.minute() < 35) pumpAstate = true;  //1:30 pm - 5 mins
  if (now.hour() == 16 && now.minute() >= 0 && now.minute() < 10) pumpAstate = true;  //4:00 pm - 10 mins
  if (pumpAstate == true)
  {
    digitalWrite(pumpA, TURN_ON);
    terminal.println("Pump A Is On");  //  Text printed to terminal monitor
  }
  else
  {
    pumpAon = 0;
    digitalWrite(pumpA, TURN_OFF);
  }
  //---------------------------Bloom B Feed Times-------------------------------------
  boolean pumpBstate = false;
  if (pumpBon == 1) pumpBstate = true;
  if (now.hour() == 18 && now.minute() >= 0 && now.minute() < 5) pumpBstate = true;     //6:00 pm - 5 mins -//- 1 hour after light on
  if (now.hour() == 20 && now.minute() >= 30 && now.minute() < 35) pumpBstate = true;   //8:30 pm - 5 mins
  if (now.hour() == 23 && now.minute() >= 0 && now.minute() < 5) pumpBstate = true;     //11:00 pm - 5 mins
  if (now.hour() == 1 && now.minute() >= 30 && now.minute() < 35) pumpBstate = true;    //1:30 am - 5 mins
  if (now.hour() == 4 && now.minute() >= 0 && now.minute() < 10) pumpBstate = true;     //4:00 am - 10 mins
  if (pumpBstate == true)
  {
    digitalWrite(pumpB, TURN_ON);
    terminal.println("Pump B Is On");  //  Text printed to terminal monitor
  }
  else
  {
    pumpBon = 0;
    digitalWrite(pumpB, TURN_OFF);
  }
  terminal.flush();
}

void ROcheck()  //RO Pump 5 gallons@170 seconds = 34 seconds per gallon
{
  if (ROpumpOn == 1 && runningRO == false)
  {
    digitalWrite(ROpump, TURN_ON);
    runningRO = true;
    ROstart = millis();
    countGallons = msPerGallon * totalGallons;
    Blynk.virtualWrite(V11, 0);
    Blynk.virtualWrite(V12, 0);
    terminal.print("Pumping:");
    terminal.print(totalGallons);
    terminal.println(" Gallons of RO");
    terminal.flush();
  }
  if (millis() - ROstart > countGallons)
  {
    ROpumpOn = 0;
    runningRO = false;
    digitalWrite(ROpump, TURN_OFF);
  }
}

void dosingPumps()
{
  if (button == 1 && pumpRunning == false)
  {
    Blynk.virtualWrite(V4, 0);
    Blynk.virtualWrite(V5, 0);
    Blynk.virtualWrite(V6, 0);
    pumpRunning = true;
    digitalWrite(pumpPin[x], HIGH);
    startPump = millis();
    runCount = DOSEml * multiplier[x];
    terminal.print("Dosing in: ");
    terminal.print(DOSEml);
    terminal.print(" milliliters of ");
    terminal.println(nuteType[x]);
  }
  if (millis() - startPump > runCount)
  {
    digitalWrite(pumpPin[x], LOW);
    pumpRunning = false;
    button = 0;
  }
  terminal.flush();
}

What do you have for pullup resistors on the I2C bus? I recently saw a problem where 47 ohm instead of 4.7K were used by accident, ended up draining the RTC battery in 24 hours.

it seems the RTC can only maintain time for at most 3 days, but it usually fails sooner.

What happens if you use just the RTC and an Arduino to display the time for a few days ? Does it still fail ?

CrossRoads:
What do you have for pullup resistors on the I2C bus? I recently saw a problem where 47 ohm instead of 4.7K were used by accident, ended up draining the RTC battery in 24 hours.

I did not know a pullup was needed for I2C, I was under the impression it was plug and play. Can you advise how to go about that, or link me to some info on it if possible? Otherwise I will read what I can on the subject.

UKHeliBob:
What happens if you use just the RTC and an Arduino to display the time for a few days ? Does it still fail ?

I haven't yet tried this, but I plugged it in last night with only the RTC connected, so I will see if it maintains past Tuesday to see. Do you have any suspicions?

Do you have any suspicions?

Only that the library may be incompatible with the hardware, but it pays to try the simplest things first.

For the record, usually, after a failure, it takes me a few days to get to the location of the project to reset the power. Usually, but not always, after reset, the time displays accurately which means the RTC is still maintaining time on battery power, or perhaps the 5V continues to supply voltage so long as the MCU is powered, regardless of whether the sketch is working properly or not. I've long felt that I need to bite the bullet and try to adopt the DS3231 library, but obviously a lot of my sketch would need to be changed, and that's what intimidates me.

Since this sounds like a device that you want to be able to recover from a lock up, I'd turn on the watchdog timer. The way it works is you set a reasonable timeout interval for the watchdog, in your loop you will be constantly resetting the timer. If for some reason your device hangs the timeout will expire and the watchdog will automatically reset the microcontroller. If there is data in SRAM that needs to persist through a reset then you need to store them in EEPROM when they are set and then load them from EEPROM on startup.

I'm not suggesting that you use the watchdog as a bandaid for faulty hardware or software. However, bugs can happen and sometimes they don't manifest themselves for some time. You should always fix them but with a device like this it's important that the system can recover from them gracefully if possible. You should code it so that resets are logged in some way so you know if there is a problem. You can actually use the watchdog as a debugging tool to determine where in your code the problem occurs. I have a library that makes this easy:

When someone says "my sketch fails at random times", the first thing I look for is String, malloc/realloc or new:

  char timeStamp[16];
  char dateStamp[16];
  sprintf(timeStamp, "%02d:%02d:%02d-%02s", displayHour, MIN, SEC, meridian);
  sprintf(dateStamp, "%02d/%02d/%04d", now.month(), now.day(), now.year());
  String ts;
  String ds;
  ts = timeStamp;
  ds = dateStamp;
  lcdA.clear();            //Advanced Mode
  lcdA.print(3, 0, ts);    //Advanced Mode
  lcdA.print(3, 1, ds);    //Advanced Mode

I seriously doubt that this limited use could be causing the problem. But it's also easy to eliminate String:

  char timeStamp[16];
  char dateStamp[16];
  sprintf(timeStamp, "%02d:%02d:%02d-%02s", displayHour, MIN, SEC, meridian);
  sprintf(dateStamp, "%02d/%02d/%04d", now.month(), now.day(), now.year());
  lcdA.clear();                //Advanced Mode
  lcdA.print(3, 0, timestamp); //Advanced Mode
  lcdA.print(3, 1, datestamp); //Advanced Mode

This saves about 1600 bytes of program space (if no other libraries are using String).

Also, you could use just one char array:

  char stamp[16];
  lcdA.clear();              //Advanced Mode
  sprintf(stamp, "%02d:%02d:%02d-%02s", displayHour, MIN, SEC, meridian);
  lcdA.print(3, 0, stamp);   //Advanced Mode
  sprintf(stamp, "%02d/%02d/%04d", now.month(), now.day(), now.year());
  lcdA.print(3, 1, stamp);   //Advanced Mode

These changes save a total of about 40 bytes of RAM, which may help avoid a stack overflow problem.

Cheers,
/dev

I've long felt that I need to bite the bullet and try to adopt the DS3231 library, but obviously a lot of my sketch would need to be changed, and that's what intimidates me.

Given that you are simply reading the time, using RTCLib.h and instantiating as RTC_DS1307 RTC; is not a problem for using a DS3231 for simple time keeping.

There are only differences when you get to the control registers and square wave, alarms, temperature compensation, nvram etc.

The rtc library is not likely to be the source of your problem.

I wish this forum had Like buttons. Thanks for all the help everyone. -dev, I updated the code as you suggested and hopefully this glitch will be resolved. Also thanks cattledog for putting me at ease regarding using the 1307 library. I was not at all looking forward to learning a new library, than updating this sketch. pert, I'm reading through your github now and it seems there is a bit more material I need to learn before I can add that in. I do wish I could log any and every failure b/c now the only way I can tell is that the Blynk app stops displaying the continuity of time. Is the consensus affirmative on pullup resistors need to be soldered to the RTC? I may already have them (resistors), but if not, I will order them immediately.

Is the consensus affirmative on pullup resistors need to be soldered to the RTC? I may already have them (resistors), but if not, I will order them immediately.

It can't hurt if it is the only i2c device on the bus. Often the internal pullups on SDA/SCl enabled by the wire library will be sufficient, but adding 4.7K resistors to the solder locations on the module will remove one issue.

I wish this forum had Like buttons.

Click that "karma [add]" link on posts that you "like". It's similar to "reputation" on stackoverflow. Click 'til it hurts!