Arduino freezes after 13 hours using ENC28J60+ArduinoJson6 library

Hi guys, I am quite new to the ethernet world. So please bear with me if I do ask stupid questions.

I designed a PCB using Atmega328P+ENC28J60 chip (schematic attached in the images below - sch1, sch2, sch3). The function of this board is basically sending GET requests to the server and retrieve a set of json data, so based on my understanding my board acts as a client only right? I am currently using UIPEthernet v2.0.9 and ArduinoJson v6.17.0, code is as below:

#include <EEPROM.h>
#include <ArduinoJson.h>
#include <UIPEthernet.h>

#define OUT0    2
#define OUT1    A3
#define OUT2    A2
#define OUT3    A1
#define OUT4    A0
#define OUT5    9
#define OUT6    8
#define OUT7    7
#define OUT8    6
#define OUT9    5
#define CS      10

// mac: 46 57 5a 6b 48 51
// mac (char): FWZkHQ
#define HOSTNAME  "autolighting.afa-sports.com"

#define ID_SIZE   6

byte mac[ID_SIZE];
char macBuffer[ID_SIZE*2];
const byte output[] PROGMEM = {OUT0, OUT1, OUT2, OUT3, OUT4, OUT5, OUT6, OUT7, OUT8, OUT9};
EthernetClient client;

void clientRead() {
  StaticJsonDocument<40> filter;
  StaticJsonDocument<120> doc;
  filter.clear();
  doc.clear();
  filter["data"]["relay_actions"] = true;
  Serial.println(doc.memoryUsage());

  client.find("\r\n\r\n");
  deserializeJson(doc, client, DeserializationOption::Filter(filter));
  client.flush();
  delay(50);

  if (!doc["data"]["relay_actions"].isNull()) {
    for (byte i = 0; i < 10; i++) {
//      Serial.print(doc["data"]["relay_actions"][i].as<bool>());
      digitalWrite(pgm_read_byte_near(&output[i]), doc["data"]["relay_actions"][i].as<bool>());
    }
//    Serial.println();
  }
  
  filter.clear();
  doc.clear();
  Serial.println(freeRam());
}

void sendReq() {
  client.println(F("GET /api/iot/master-controller/get-command HTTP/1.1"));
  client.println(F("Host: autolighting.afa-sports.com"));
  client.println(F("DEVICE-ID: 46575a6b4851"));
//  client.print(F("DEVICE-ID: "));
//  client.println(macBuffer);
  client.println(F("Connection: close"));
  client.println();
}

void setup() {
  Serial.begin(115200);

  for (byte i = 0; i < 10; i++) {
    pinMode(pgm_read_byte_near(&output[i]), OUTPUT);
    digitalWrite(pgm_read_byte_near(&output[i]), LOW);
  }
  
  for (uint8_t i = 0; i < ID_SIZE; i++) {
    byte charByte = EEPROM.read(i);
    if (charByte != 0) {
      char temp[2];
      mac[i] = charByte;
      itoa(mac[i], temp, 16);
      strcat(macBuffer, temp);
      free(temp);
    }
  }

  Serial.println(macBuffer);

  Ethernet.init(CS);
  while (!Ethernet.begin(mac)) {
    Serial.println(F("Failed to configure Ethernet"));
  }
  delay(1000);
}

void loop() {
  Ethernet.maintain();
  if (client.connect(HOSTNAME, 80)) {
    sendReq();
    clientRead();
  }
  client.stop();
  delay(2000);
}

Due to the high SRAM consumption (1.3kb) and I might have other things (not sure what yet) to add in to the board in the future, I tried to minimize the dynamic memory by changing this (in the uipethernet-conf.h file):

#define UIP_SOCKET_NUMPACKETS    5
#define UIP_CONF_MAX_CONNECTIONS 4
#define UIP_CONF_UDP_CONNS       4

to this (888bytes SRAM):

#define UIP_SOCKET_NUMPACKETS    2
#define UIP_CONF_MAX_CONNECTIONS 1
#define UIP_CONF_UDP_CONNS       1

I'm wondering will this affect the system performance? Btw, I've also set the timeout to 10s

After 13 hours of smooth operation, the board freezes and only became normal when i hard-reset the board. Currently, I'm connecting my board to a wireless extender, because I am not sitting right next to the wifi router.

So basically my question would be what would be the problem causing the board to freeze? This piece of code below is my previous version, but the board freezes even quicker (around 3hours):

#include <EEPROM.h>
#include <ArduinoJson.h>
#include <UIPEthernet.h>

#define OUT0    2
#define OUT1    A3
#define OUT2    A2
#define OUT3    A1
#define OUT4    A0
#define OUT5    9
#define OUT6    8
#define OUT7    7
#define OUT8    6
#define OUT9    5
#define CS      10

// host name: "autolighting.afa-sports.com
// resource: "/api/iot/master-controller/get-command"
// mac: 46 57 5a 6b 48 51
// mac (char): FWZkHQ
#define HOSTNAME  "autolighting.afa-sports.com"

#define ID_SIZE   6

byte mac[ID_SIZE];
const byte output[] PROGMEM = {OUT0, OUT1, OUT2, OUT3, OUT4, OUT5, OUT6, OUT7, OUT8, OUT9};
EthernetClient client;
StaticJsonDocument<40> filter;
StaticJsonDocument<120> doc;

void clientRead() {
  filter.clear();
  doc.clear();
  filter["data"]["relay_actions"] = true;

  deserializeJson(doc, client, DeserializationOption::Filter(filter));
  client.flush();

  if (!doc["data"]["relay_actions"].isNull()) {
    for (byte i = 0; i < 10; i++) {
//      Serial.print(doc["data"]["relay_actions"][i].as<bool>());
      digitalWrite(pgm_read_byte_near(&output[i]), doc["data"]["relay_actions"][i].as<bool>());
    }
  }
}

void serialRead() {
  Serial.println(F("ACK"));
  unsigned long prevTime = millis() + 3000;
  while (millis() < prevTime) {
    while (Serial.available()) {
      while (true) {
        StaticJsonDocument<100> doc;
        deserializeJson(doc, Serial);
        if (doc["Read"] == true) {
//          for (byte i = 0; i < 6; i++) doc["ID"].add(mac[i]);
//          serializeJson(doc, Serial);
          
          Serial.print(F("{\"ID\":["));
          for (byte i = 0; i < ID_SIZE; i++) {
            Serial.print(mac[i]);
            if (i != ID_SIZE-1) Serial.print(F(","));
          }
          Serial.println(F("]}"));
        }
        else if (doc["SetID"] == true) {
          // use atoi to convert char array to int
          // then write to eeprom
          memset(mac, 0, ID_SIZE);
          for (byte i = 0; i < ID_SIZE; i++) mac[i] = doc["ID"][i].as<byte>();
          for (byte i = 0; i < ID_SIZE; i++) EEPROM.write(i, 0);
          for (byte i = 0; i < ID_SIZE; i++) EEPROM.write(i, mac[i]);
        }
      }
    }
  }
}

void connectToServer() {
  if (!client.connect(HOSTNAME, 80)) {
    Serial.println(F("Connection failed"));
    return;
  }
  delay(100);

  client.println(F("GET /api/iot/master-controller/get-command HTTP/1.1"));
  client.println(F("Host: autolighting.afa-sports.com"));
  client.println(F("DEVICE-ID: 46575a6b4851"));
  client.println(F("Connection: close"));
  client.println();
  delay(500);

  client.find("\r\n\r\n");
  if (client.available()) clientRead();
  if (!client.connected()) client.stop();
}

void setup() {
  Serial.begin(115200);

  for (byte i = 0; i < 10; i++) {
    pinMode(pgm_read_byte_near(&output[i]), OUTPUT);
    digitalWrite(pgm_read_byte_near(&output[i]), LOW);
  }
  
  for (uint8_t i = 0; i < ID_SIZE; i++) {
    byte charByte = EEPROM.read(i);
    if (charByte != 0) {
      mac[i] = charByte;
    }
  }

  serialRead();

  Ethernet.init(CS);
  while (!Ethernet.begin(mac)) {
    Serial.println(F("Failed to configure Ethernet"));
  }
  client.setTimeout(10000);
  delay(1000);
}

void loop() {
  Ethernet.maintain();
  connectToServer();
  delay(2000);
}

I'm sorry if I missed out any important information. Your help is greatly appreciated. Thank you very much.

Just a quick guess but I see this type of operation when the stack runs out of memory and overwrites other goodies. It drives people bonkers. The hint you gave is it runs for a while then bombs indicating the code is probably OK. Check for something that uses RAM like building strings etc. I am assuming the code change uses more RAM. Check to be sure your regulators are not getting hot. I did not take the time to go through the code, I just did a quick look and nothing jumped out at me. Try and add a few lines of code and dump the number of bytes of free RAM every so often, if it keeps shrinking you know what the problem is, the hard part will be to find it.

which version of UIPEthernet do you use? I made many fixes and improvements in the current version.
and you could try my EthernetENC library

gilshultz:
Just a quick guess but I see this type of operation when the stack runs out of memory and overwrites other goodies. It drives people bonkers. The hint you gave is it runs for a while then bombs indicating the code is probably OK. Check for something that uses RAM like building strings etc. I am assuming the code change uses more RAM. Check to be sure your regulators are not getting hot. I did not take the time to go through the code, I just did a quick look and nothing jumped out at me. Try and add a few lines of code and dump the number of bytes of free RAM every so often, if it keeps shrinking you know what the problem is, the hard part will be to find it.

Hi @gilshultz, I suspected that as well, in my previous version of codes, I included the freeRam function as below:

int freeRam () {    // check remaining RAM space
  extern int __heap_start, *__brkval;
  int v;
  return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}

It returned me a constant amount of RAM available (which is 587, I tried it quite sometime ago, couldn't remember that), and after running for around 3 hours, it suddenly gave me negative values. I was stunned and I saw in a forum, people saying negative value means the program ran out of memory. However, I was curious about: Wouldn't the arduino restart itself when it fully uses up its SRAM?

Back to the topic, then I updated the configuration file with the settings mentioned earlier, the RAM usage function returned me 933 constantly, with no more negative values (that's a good sign I think). But it still freezes after a good 13 hours or so.

Juraj:
which version of UIPEthernet do you use? I made many fixes and improvmentes in the current version.
and you could try my EthernetENC library

Hi @Juraj, I'm currently using UIPEthernet v2.0.9 (mentioned in my post). I noticed that library as well, developed by the same author of the UIPEthernet, not sure what's the difference between the two, I will definitely try that now.
I just tried EtherCard and it really cracked my head, it's not as user friendly as UIPEthernet.

IIRC, the json library uses String objects which are a curse on any Arduino with limited RAM. If they do cause you to run out of memory, there's no reason to expect an automatic restart.

wildbill:
IIRC, the json library uses String objects which are a curse on any Arduino with limited RAM. If they do cause you to run out of memory, there's no reason to expect an automatic restart.

Which means other than watchdog and brownout detection will trigger reset to the Arduino will not trigger automatic restart right? Thanks for the info.

wildbill:
IIRC, the json library uses String objects which are a curse on any Arduino with limited RAM. If they do cause you to run out of memory, there's no reason to expect an automatic restart.

it doesn't use String

Juraj:
it doesn't use String

Last time I looked at an Arduino JSON library's source it was riddled with them. Whether it was this particular library though, I don't recall.

Juraj:
it doesn't use String

I was intrigued enough to go & grep the source in the library I have on my machine. I stand corrected: not a String object in sight.

Hi guys, an update here, after using @juraj 's EthernetENC library, the arduino still freezes after smooth operation of around 13 hours. Then, I checked the UIPClient::connect function, the timeout checking is written:

int32_t timeout = millis() + 1000 * UIP_CONNECT_TIMEOUT;
...
if (((int32_t)(millis() - timeout)) > 0)
{
    conn->tcpstateflags = UIP_CLOSED;
    break;
}

Just wondering is it possible to have this kind of scenario where the millis() rolled-over, causing it to set to negative value (since it is not declared as uint32_t). As compared to the conventional Ethernet library (for Wiznet chips), the connect function uses uint32_t instead:

uint32_t start = millis();
while (1) {
    uint8_t stat = Ethernet.socketStatus(sockindex);
    if (stat == SnSR::ESTABLISHED) return 1;
    if (stat == SnSR::CLOSE_WAIT) return 1;
    if (stat == SnSR::CLOSED) return 0;
    if (millis() - start > _timeout) break;
    delay(1);
}

I know they are both different libraries for different ethernet chips, but the timeout feature should have the serve the same purpose right?

Please correct me if I have missed out something. I will change the int32_t to uint32_t and try again tonight.

the timeout check was written this way by the original author of the library. I don't like it but it works right so I didn't change it

Is it possible that the server has some settings that only allows a client to connect for a certain period of time only? Because I've reduced the memory usage from 60% to 46%, but the amount of time taken to freeze the arduino is still the same. That leads me to a suspicion that it isn't a memory issue.

What do you think about that?

my fairly complex Regulator project now runs more then 24 hours without problems with EthernetENC on ATmega1284p

Hey @juraj, I will definitely go on with using ATmega2560 if I have the chance, because it has plenty of SRAM and flash. However, I have already designed the PCB using ATmega328P, so I was thinking is there any possible way to maintain that chip while able to carry out the operations smoothly.

txlim96:
Hey @juraj, I will definitely go on with using ATmega2560 if I have the chance, because it has plenty of SRAM and flash. However, I have already designed the PCB using ATmega328P, so I was thinking is there any possible way to maintain that chip while able to carry out the operations smoothly.

it didn't meant it as suggestion to use a MCU with more memory. my project uses 1284p, but I don't think it is relevant here if there is enough memory for the project on the 328p