HttpClient post failes after looping a few times

tldr;
I feel like there is a memory issue with HttpClient.post as it seems to corrupt the data and then fail.

I am working on a project that gets device settings and then display content to display and have a few classes to manage the system, settings, and display. Please bare with me as this is my first post in the forums and I'll try to add in everything related to the process, but I've taken out the private bits and left the relevant areas.

void loop() {
  //var
  DynamicJsonDocument hardwareQuery(512);
  //var

  myClass.connectToWiFi();

      hardwareQuery = myClass2.hardware_checkin();

      if(hardwareQuery["approved"] | false)
      {
        myClass3.process_approved(hardwareQuery, false, false);
        myClass.putToSleep(10000);
      } else {
        myClass3.display_code(hardwareQuery);
        myClass.putToSleep(5000); //5 seconds
      }
}

The connectToWiFi is fairly simple

void myClass::connectToWiFi(void)
{
  while (WiFi.status() != WL_CONNECTED) {
    Serial.println("Attempting to connect to SSID: " + String(deviceSettings.wifiSsid));

    WiFi.begin(deviceSettings.wifiSsid, deviceSettings.wifiPassword);

    printWiFiStatus(WiFi.status());
    if(WiFi.status() == WL_CONNECTED)
    {
      Serial.print("Signal strength in dBm: ");
      Serial.println(WiFi.RSSI());
    }
  }

  Serial.println("Connected to SSID: " + String(deviceSettings.wifiSsid));
}

Then there is the post function that ends up failing

DynamicJsonDocument myClass2::hardware_checkin(void)
{
	//var
  WiFiSSLClient wifiClient;
  HttpClient httpClient = HttpClient(wifiClient, "domain.com", 443);
  String responseBody = "";
  unsigned short postResponse = 0;
  //var
  
  httpClient.setTimeout(5000);
  httpClient.setHttpResponseTimeout(5000);
  postResponse = httpClient.post("/path", "application/json", "{ \"json\": \"object\" }");
  responseBody = httpClient.responseBody();
  httpClient.flush();
  httpClient.stop();
  wifiClient.flush();

  Serial.println(responseBody);
  Serial.println("postResponse: " + String(postResponse));
}

As you can see where the printlns are, and when she's running happily, I get a clean output of

---=== Wake ===---
Connected to SSID: <ssid>
{"success":true,"executionTime":487}
postResponse: 0
{"success":"true","executionTime":466}
---=== Sleep for 10 sec ===---

As time passes, it has to reconnect to the WiFi more regularly

---=== Wake ===---
Attempting to connect to SSID: <ssid>
WiFi status: connected to WiFi
Signal strength in dBm: -23
Connected to SSID: <ssid>
{"success":true,"executionTime":511}
postResponse: 0
{"success":"true","executionTime":708}
---=== Sleep for 10 sec ===---

Then httpClient.responseBody() from httpClient.post stops returning just the body and starts to return the entire response. Note the 5 second sleep here vs the 10 second, that's because the "body" is no longer just the JSON object.

---=== Wake ===---
Attempting to connect to SSID: <ssid>
WiFi status: connected to WiFi
Signal strength in dBm: -35
Connected to SSID: <ssid>
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 199
Connection: close
Date: Tue, 24 Oct 2023 17:08:25 GMT
x-amzn-RequestId: <redacted>
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token
x-amz-apigw-id: <redacted>
Access-Control-Allow-Methods: POST,OPTIONS
X-Amzn-Trace-Id: <redacted>
X-Cache: Miss from cloudfront
Via: 1.1 <redacted> (CloudFront)
X-Amz-Cf-Pop: <redacted>
X-Amz-Cf-Id: <redacted>

{"success":true,"executionTime":527}
postResponse: 0
---=== Sleep for 5 sec ===---

Then everything just stops and the post response code is maxint and everything just loops and HttpClient seems to be dead.

---=== Wake ===---
Connected to SSID: <ssid>

postResponse: 65535
---=== Sleep for 5 sec ===---

I can see that the post is still being made though the firewall.

However, it looks like they are not getting to the server code, as the logs for it ended about an hour ago.

Not getting to the server could be a number of things as it is only accepting POST and a couple of other items, so this could be a number lets say "proxy" items or just a simple corrupted request that cannot be processed.
I'll also note that other devices that are making these same API calls (Ras Pi, Ubuntu, and Windows) are not exhibiting any issues accessing the APIs and getting the data so this issue is related to the Arduino.

My guess is memory management.
So, as this project moves forward, is there a way to

  1. Read what memory is available so that I can look for leaks and know what space I have available?
  2. Defrag the memory so that objects fit?
  3. Most of these are local vars, but maybe the WiFiSSLClient or HttpClient classes stay resident and need more of a manual destroy?

I do hope this is enough detail to understand what's happening and someone can give some thought into where the issue might be.

First things first: exactly, which Arduino model are you using?

And, even if I can't bet the problem is this (because I don't know if you're using a UNO, and I can see just a couple of functions from your code), do you know "String" class variables are generally deprecated due to memory fragmentation (Arduino has no garbage collector or memory defrag)? I'd start by converting all "String" variables into standard, pre-allocated "C strings", starting from "responseBody ".

I'm sorry, I was too focused on the code and forgot the hardware.
I'm working with a mkr1010 for this project.
I doubt that it matters but I do have a 2000mAh li-po connected to it.

As I was working on trying to tighten up the memory use last night, I remember using memory profilers back when I worked in C and C++ more, are there any tools that can do that for this device?

There are a few hurdles with memory that I have not gotten to yet like display buffers and firmware updates, and I'm not really sure what the resource use is. Maybe there is an emulator that can provide that resource useage?

Ok, in this case I'm sorry but I can't be of much help, I never worked with MKR.

But as I suspect it's a MKR board specific issue/question, I suggest you to move your post to MKRWiFi1010 area (or ask a moderator to do it for you) where you'll find more focused answers.

PS: even if I never worked with MKR, I still think the questions you made here have no answers at all or can't help you much, because even if MKR1010 has a more powerful processor and a lot of available memory compared with UNO, I suspect there's no high level memory management like the one you're asking for...

Thank you for your feedback.

Relating to the String class, I was not aware of the issues there. I was seeing demos and it being used and just I'm rusty on my C/C++ since the last time I was actively developing in it was ~30 years ago.
I'll convert those back to nice safe char arrays.

The infamous "String problem" is critical on UNO for sure because it has little RAM available, and can cause problems ranging from unexpected behaviours up to complete code lock due to memory fragmentation.
But it depends on how many String variables there are in the code, how large they are, and how many changes are made to their contens. If you have a few "String" variables, and/or have a constant value, and/or the changes are limited, you could use them even on a UNO.
You can find more detailed info about Strings and fragmentation HERE.

On boards with larger memory the issue could be very unfrequent up to even never seen, but that's a bit like sweeping the dust under the carpet: unless some kind of garbage collector and/or an allocation manager are somehow implemented, the memory can still be affected by fragmentation problems coming up in the future.

That's why I always prefer "char arrays" (aka "C strings") with preallocated size/memory, because it lets you in control over memory.

Please note: all this is just my personal opinion based on a few years working with UNO, Mega, ESP8266, etc. boards so I don't know if some of the newer ones like UNO R4 or ESP32 based, and including MKR obviously, solved this problem (I'm open to know more about that).

I understand and agree.
And, if we're writing C/C++, what are we doing if not keeping memory clean? I guess I'm not used to bad public libraries yet. And wow, yea 2K of memory really keeps you on your toes on the Uno, doesn't it?

As I was converting everything away from String, I did find the root of where I moved to them. It was the "ArduinoHttpClient" library that is "by" Arduino GitHub - arduino-libraries/ArduinoHttpClient: Arduino HTTP Client library, that is where all of the String examples started and I moved over to. Even though this is "experimental" it is currently maintained by Arduino (as of 3 months ago).
No memory warnings, cautions, or anything. There are a ton of posts that uses that to be able to do a POST easier.
There really should be a note "hay, this is to start but it kills memory so please move away later" or something that discusses the String/memory concerns.

My problem with memory is what we are building can go offline for quite a while so we are trying to store some data for use later.
Think "directions", you connect to an API when you are at the office and get a route and store that locally.
Then you run on battery for a few days and maybe every hour it wakes up, checks a GPS, and updates the display with the next step.
Basically, we won't have access to be able to do updates every hour so we need to cache it, so I'm glad that I have my 32K for that cache.
--That is not the actual use case but it's the same basic idea.

That being said, and clearly you know what you're talking about and have been living with 2K for a while....
You are going to tell me that the "Arduinojson" library kills memory as well, aren't you? That I'm going to have to make a custom class that parses out the JSON from a c-string into custom classes with basic objects, arrays, and data types and build all my own comparisons and everything to keep the memory low and clear, aren't you?
Even when I was working with C, I did have a good 4M of RAM to work with so I was careful on memory use but not 32K careful and I also wasn't making network calls or trying to use JSON return data from those calls so the data that I was moving was already fairly small.

In general, are there any pitfalls (libraries) to stay away from since the String/ArduinoHttpClient does not have warnings?
I'm fairly sure that I'm still overflowing somewhere and I'm trying to debug where that is while keeping enough allocated for things to actually work, but I'm still struggling to know what I have to work with when there doesn't seem to be a way to actually measure resources. If I could, I could sit and go "crap, 2k left, I need to split this, de-allocate that, or change char allocations. Maybe the API even needs to be thinned out but that would effect other devices calling it as well.
Right now, when the Arduino just stops and I'm guessing it's memory.

I remember using the MS Visual C++ memory profiler.... something when we were building larger systems to ensure everything fit, I was really hoping that some IDE or internal print() could provide that knowledge/insight.
I still don't even have production data back from the APIs yet so I'm not even 100% positive on the max size yet.

OH!
So, is there a way to try/catch a memory overflow instead of having the Ardiuno die? At least then I could go into an error state and maybe write out some notes on where the error state happened.

Alright, I thinned down the code and collapsed the 3 classes into a single sketch.
Yes, some of the code doesn't make any sense right now like the "firstBoot" but this is code that is being ported from another system and we are not done with that move.
Additionally, yes, the if in the main loop is dumb but more goes in there that was removed. I left in the if just to leave the parts that gets to the easiest fail case.

I've marked the section where the Arduino evetually dies.

I did leave in enough so that the code will run and reaturn valid data from the API for any testing.

/**
 *  @filename   :   
 *  @brief      :   
 *  @author     :   
 *  @Libraries  :
 *    FlashStorage@1.0.0
 *    Arduinojson@6.21.3
 *    WiFiNINA@1.4.12
 *    ArduinoHttpClient@0.5.0
 *    RTCZero@1.6.0
 *    Arduino Low Power@1.2.2
 *    BQ24195@0.9.2
 *    Time@1.6.1
 *    NTPClient@3.2.1
 *
 *  Copyright (C) Barr Technologies 2023/10/18
 */

#include <SPI.h>
#include <ArduinoJson.h>
#include <BQ24195.h>
#include <FlashStorage.h> 
#include <WiFiNINA.h>
#include <TimeLib.h>
#include <NTPClient.h>
#include "ArduinoLowPower.h"

typedef struct { 
  char hostname[256];
  char wifiSsid[32];
  char wifiPassword[32];
} DeviceSettings;


//var
bool debugSerial = true;
bool debugLed = true;
bool deviceApproved = false;
bool firstBoot = false;
inline static DeviceSettings deviceSettings;
inline static bool classDebugSerial = false;
inline static bool classDebugLed = false;
inline static bool usedFailback = false;
inline static char macAddress[18] = "";
inline static StaticJsonDocument<256> previousHardwareResults;
//var

#define RED_LED     25
#define GREEN_LED   26
#define BLUE_LED    27
#define FAILSAFE1   "failoverssid"
#define FAILSAFE2   "failoverpass"

FlashStorage(deviceSettings_flashStore, DeviceSettings); 

void setup() {
  if(debugSerial)
  {
      delay(5000); // 5 seconds
      Serial.begin(115200);
      Serial.println(F("---=== Booted ===---"));
  }

  systemInit(debugSerial, debugLed);
  hardwareInit(debugSerial, debugLed);

  firstBoot = true;
}

void loop() {

  connectToWiFi();

  if(firstBoot)
  {
    firstBoot = false;
  }

  hardware_checkin();

  if(previousHardwareResults["approved"] | false)
  {
    putToSleep(5000);
  } else {
    display_authCode(previousHardwareResults);
    putToSleep(5000); //5 seconds
  }
}

void systemInit(bool& debugSerial, bool& debugLed)
{
  //var
  bool updateFlash = false;
  //var

  Serial.println(F("Pre flash"));
  Serial.println(deviceSettings.hostname);
  Serial.println(deviceSettings.wifiSsid);
  Serial.println(deviceSettings.wifiPassword);

  classDebugSerial = debugSerial;
  classDebugLed = debugLed;

  WiFiDrv::pinMode(RED_LED, OUTPUT);
  WiFiDrv::pinMode(GREEN_LED, OUTPUT);
  WiFiDrv::pinMode(BLUE_LED, OUTPUT);

  WiFi.lowPowerMode();

  deviceSettings = deviceSettings_flashStore.read();

  if(strlen(deviceSettings.hostname) == 0)
  {
    get_hostname();
    updateFlash = true;
  }

  if(strlen(deviceSettings.wifiSsid) == 0 || strlen(deviceSettings.wifiPassword) == 0)
  {
    strcpy(deviceSettings.wifiSsid, "ssid");
    strcpy(deviceSettings.wifiPassword, "pass");

    updateFlash = true;
  }

  if(updateFlash && !classDebugSerial)
    deviceSettings_flashStore.write(deviceSettings);

  Serial.println(F("Post flash"));
  Serial.println(deviceSettings.hostname);
  Serial.println(deviceSettings.wifiSsid);
  Serial.println(deviceSettings.wifiPassword);
}

void get_hostname(void)
{
  //var
  char hexChar[2];
  byte mac[6];
  //var

  WiFi.macAddress(mac);

  strcpy(deviceSettings.hostname, "redacted");
  for (int runner = 2; runner >= 0; runner--) {
    sprintf(hexChar, "%02x", mac[runner]);
    strcat(deviceSettings.hostname, hexChar);
  }
}

void connectToWiFi(void)
{
  //var
  WiFiUDP wifiClient;
  NTPClient timeClient(wifiClient);
  unsigned short errorCount = 1;
  //var

  while (WiFi.status() != WL_CONNECTED) {
    if(classDebugLed)
    {
      WiFiDrv::analogWrite(GREEN_LED, 1);
      WiFiDrv::analogWrite(RED_LED, 1);
    }

    if(classDebugSerial)
    {
      Serial.print(F("WiFi join attempt: "));
      Serial.println(errorCount);
    }

    if(!usedFailback)
    {
      if(classDebugSerial)
      {
        Serial.print(F("Attempting to connect to SSID: "));
        Serial.println(deviceSettings.wifiSsid);
      }

      WiFi.begin(deviceSettings.wifiSsid, deviceSettings.wifiPassword);
    }
    else
    {
      if(classDebugSerial)
      {
        Serial.print(F("Attempting to connect to failsafe SSID: "));
        Serial.println(FAILSAFE1);
      }
      
      WiFi.begin(FAILSAFE1, FAILSAFE2);
    }

    if(classDebugSerial)
      printWiFiStatus(WiFi.status());

    if(WiFi.status() == WL_CONNECTED)
    {
      if(classDebugSerial)
      {
        Serial.print(F("Signal strength in dBm: "));
        Serial.println(WiFi.RSSI());
      }

      if(timeStatus() == timeNotSet)
      {
        timeClient.begin();
        timeClient.setTimeOffset(0);
        timeClient.update();
        setTime(timeClient.getEpochTime());

        if(classDebugSerial)
          Serial.println(F("Clock synced to NTP"));
      }
    } else {
      if(classDebugLed)
      {
        WiFiDrv::analogWrite(GREEN_LED, 0);
        WiFiDrv::analogWrite(RED_LED, 1);
      }

      if(errorCount++ >= 6) //looped 6 times for 10 seconds each time for 1 min total delay
        usedFailback = true;

      if(classDebugSerial)
      {
        Serial.println(F("---=== Slow down WiFi join ===---"));
        delay(10000);
        Serial.println(F("---=== Retry WiFi ===---"));
      }
      else
        LowPower.deepSleep(10000);
    }

  }

  if(classDebugSerial)
  {
    Serial.print(F("Connected to SSID: "));
    if(!usedFailback)
      Serial.println(deviceSettings.wifiSsid);
    else
      Serial.println(FAILSAFE1);
  }

  if(classDebugLed)
  {
    WiFiDrv::analogWrite(GREEN_LED, 0);
    WiFiDrv::analogWrite(RED_LED, 0);
  }
}

void printWiFiStatus(int wifiStatus)
{
  Serial.print(F("WiFi status: "));
  switch (wifiStatus) {
    case WL_CONNECTED: Serial.println(F("connected to WiFi")); break;
    case WL_AP_CONNECTED : Serial.println(F("connected in Access Point mode")); break;
    case WL_AP_LISTENING : Serial.println(F("listening for connections in Access Point mode")); break;
    //case WL_NO_SHIELD: Serial.println(F("assigned when no WiFi shield is present;")); break;
    case WL_NO_MODULE: Serial.println(F("communication with an integrated WiFi module fails")); break;
    case WL_IDLE_STATUS: Serial.println(F("temporary status assigned when WiFi.begin() is called and remains active until the number of attempts expires (resulting in WL_CONNECT_FAILED) or a connection is established (resulting in WL_CONNECTED)")); break;
    case WL_NO_SSID_AVAIL: Serial.println(F("no SSID is available")); break;
    case WL_SCAN_COMPLETED: Serial.println(F("scan networks is completed")); break;
    case WL_CONNECT_FAILED: Serial.println(F("connection fails for all the attempts")); break;
    case WL_CONNECTION_LOST: Serial.println(F("connection is lost")); break;
    case WL_DISCONNECTED: Serial.println(F("disconnected from network")); break;
  }
}

void hardware_checkin(void)
{
  //var
  WiFiSSLClient wifiClient;
  char rssi_string[5] = "";
  char lanIP[16] = "";
  char wanIP[16] = "";
  char postBody[256] = "";
  char responseBody[512] = "";
  StaticJsonDocument<512> responseJson;
  char previousData_string[256] = "";
  char responseData_string[256] = "";
  //var

  if(classDebugLed)
    WiFiDrv::analogWrite(GREEN_LED, 1);
  
  sprintf(rssi_string, "%d", WiFi.RSSI());
  get_lanip_address(lanIP);
  get_wanip_address(wanIP);

  strcpy(postBody, "{ \"macAddress\": \"");
  strcat(postBody, macAddress);
  strcat(postBody, "\", \"lanip\": \"");
  strcat(postBody, lanIP);
  strcat(postBody, "\", \"wanip\": \"");
  strcat(postBody, wanIP);
  strcat(postBody, "\", \"resolution\": \"");
  strcat(postBody, "x");
  strcat(postBody, "\", \"hostname\": \"");
  strcat(postBody, deviceSettings.hostname);
  strcat(postBody, "\", \"lowMem\": ");
  strcat(postBody, "true");
  strcat(postBody, ", \"rssi\": ");
  strcat(postBody, rssi_string);
  strcat(postBody, ", \"battery\": ");
  strcat(postBody, "0");
  strcat(postBody, " }");

  network_securePost("production.wolfeborokeene.com", "/display", "application/json", postBody, responseBody);
  //delete[] postBody;  <==-- THIS KILLS THE DEVICE

  //ARDUINO EVENTUALLY CRASHES HERE, AFTER THE POST AND BEFORE THE PRINTLN
  //ARDUINO EVENTUALLY CRASHES HERE, AFTER THE POST AND BEFORE THE PRINTLN
  //ARDUINO EVENTUALLY CRASHES HERE, AFTER THE POST AND BEFORE THE PRINTLN
  //ARDUINO EVENTUALLY CRASHES HERE, AFTER THE POST AND BEFORE THE PRINTLN
  //ARDUINO EVENTUALLY CRASHES HERE, AFTER THE POST AND BEFORE THE PRINTLN

  if(classDebugSerial)
    Serial.println(responseBody);

  deserializeJson(responseJson, responseBody);
  delete[] responseBody;

  if(classDebugSerial)
  {
    Serial.print("responseJson size: "); 
    Serial.println(responseJson.memoryUsage()); 
  }

  previousHardwareResults.remove("updatedSettings");

  if(responseJson.containsKey("data"))
  {
    serializeJson(responseJson["data"], responseData_string);
    serializeJson(previousHardwareResults, previousData_string);

    if(strcmp(previousData_string, responseData_string) == 0)
      previousHardwareResults["updatedSettings"] = false;
    else {
      previousHardwareResults = responseJson["data"];
      previousHardwareResults["updatedSettings"] = true;
    }

    delete[] responseData_string;
    delete[] previousData_string;
  } else
    previousHardwareResults["updatedSettings"] = false;

  responseJson.clear();
  responseJson.garbageCollect();

  if(classDebugLed)
    WiFiDrv::analogWrite(GREEN_LED, 0);
}

void get_lanip_address(char ipAddress[16])
{
  //var
  IPAddress lanIPObject = WiFi.localIP();
  char ipOctet[3];
  //var

  sprintf(ipOctet, "%d", lanIPObject[0]);
  strcat(ipAddress, ipOctet);
  strcat(ipAddress, ".");
  sprintf(ipOctet, "%d", lanIPObject[1]);
  strcat(ipAddress, ipOctet);
  strcat(ipAddress, ".");
  sprintf(ipOctet, "%d", lanIPObject[2]);
  strcat(ipAddress, ipOctet);
  strcat(ipAddress, ".");
  sprintf(ipOctet, "%d", lanIPObject[3]);
  strcat(ipAddress, ipOctet);
}

void get_wanip_address(char ipAddress[16])
{
  network_get("api.ipify.org", "/", ipAddress);
}

void putToSleep(int ms)
{
  if(classDebugSerial)
  {
    Serial.print(F("---=== Sleep for "));
    if(ms < 1000) {
      Serial.print(ms);
      Serial.print(F(" ms"));
    } else if(ms < 60000) {
      Serial.print(ms / 1000);
      Serial.print(F(" seconds"));
    } else if(ms < 3600000) {
      Serial.print(ms / 60000);
      Serial.print(F(" minutes"));
    } else {
      Serial.print(ms / 3600000);
      Serial.print(F(" hours"));
    };
      Serial.println(F(" ===---"));
  }

  if(ms > 60000)
  {
    //If sleeping more then 1 min then disconnect
    Serial.println(F("Disconnect WiFi"));
    WiFi.disconnect();
    WiFi.end();
  }

  if(classDebugSerial)
  {
    delay(ms);
    Serial.println(F("---=== Wake ===---"));
  }
  else
    LowPower.deepSleep(ms);
}

void network_get(char domain[], char url[], char body[])
{
  //var
  WiFiClient wifiClient;
  unsigned long requestTimeout = millis() + 5000;
	short headerMarker = 0;
  short responseCharacter = 0;
  unsigned int bodyRunner = 0;
  //var

  if (!wifiClient.connect(domain, 80)) {
    if(classDebugSerial)
    {
      Serial.print(F("Failed to connect to "));
      Serial.println(domain);
    }
  } else {
    wifiClient.print(F("GET "));
    wifiClient.print(url);
    wifiClient.println(F(" HTTP/1.1"));
    wifiClient.print(F("Host: "));
    wifiClient.println(domain);
    wifiClient.println(F("Connection: close"));
    wifiClient.println();

    while (wifiClient.available() == 0) {
      if (requestTimeout - millis() < 0) {
        if(classDebugSerial)
          Serial.println(F("get request Timeout"));
        wifiClient.stop();
        return;
      }
    }

    while (wifiClient.connected() && headerMarker != 4)
    {
      responseCharacter = wifiClient.read();
      if (responseCharacter > 0)
      {
        //The end of the header is *defined* as an empty line, the sequence "\r\n\r\n".
        switch (headerMarker)
        {
          case 0:
          case 2:
            if (responseCharacter == '\r') headerMarker++; else headerMarker = 0;
            break;

          case 1:
          case 3:
            if (responseCharacter == '\n') headerMarker++; else headerMarker = 0;
            break;
          }
      }
    }

    if (headerMarker == 4)
    {
      //Found blank line, everything else is payload.
      while (wifiClient.available())
        body[bodyRunner++] = wifiClient.read();
      body[bodyRunner] = '\0';
    }
  }

  wifiClient.flush();
  wifiClient.stop();
}

void network_securePost(char domain[], char url[], char contentType[], char postBody[], char  body[])
{
  //var
  WiFiSSLClient wifiClient;
  unsigned long requestTimeout = millis() + 5000;
	short headerMarker = 0;
  short responseCharacter = 0;
  unsigned int bodyRunner = 0;
  //var

  if (!wifiClient.connect(domain, 443)) {
    if(classDebugSerial)
    {
      Serial.print(F("Failed to connect to "));
      Serial.println(domain);
    }
  } else {
    wifiClient.print(F("POST "));
    wifiClient.print(url);
    wifiClient.println(F(" HTTP/1.1"));
    wifiClient.print(F("Host: "));
    wifiClient.println(domain);
    wifiClient.print(F("Content-type: "));
    wifiClient.println(contentType);
    wifiClient.print(F("Content-Length: "));
    wifiClient.println(strlen(postBody));
    wifiClient.println(F("Connection: close"));
    wifiClient.println();
    wifiClient.println(postBody);

    while (wifiClient.available() == 0) {
      if (requestTimeout - millis() < 0) {
        if(classDebugSerial)
          Serial.println(F("secure post request Timeout"));
        wifiClient.stop();
        return;
      }
    }

    while (wifiClient.connected() && headerMarker != 4)
    {
      responseCharacter = wifiClient.read();
      if (responseCharacter > 0)
      {
        //The end of the header is *defined* as an empty line, the sequence "\r\n\r\n".
        switch (headerMarker)
        {
          case 0:
          case 2:
            if (responseCharacter == '\r') headerMarker++; else headerMarker = 0;
            break;

          case 1:
          case 3:
            if (responseCharacter == '\n') headerMarker++; else headerMarker = 0;
            break;
          }
      }
    }

    if (headerMarker == 4)
    {
      //Found blank line, everything else is payload.
      while (wifiClient.available())
        body[bodyRunner++] = wifiClient.read();
      body[bodyRunner] = '\0';
    }
  }

  wifiClient.flush();
  wifiClient.stop();
}

void hardwareInit(bool debugSerial, bool debugLed)
{
  classDebugSerial = debugSerial;
  classDebugLed = debugLed;

  WiFiDrv::pinMode(RED_LED, OUTPUT);
  WiFiDrv::pinMode(GREEN_LED, OUTPUT);
  WiFiDrv::pinMode(BLUE_LED, OUTPUT);

  get_mac_address();
}

void get_mac_address(void)
{
  //var
  char hexChar[2];
  byte mac[6];
  //var

  WiFi.macAddress(mac);

  for (int runner = 5; runner >= 0; runner--) {
    sprintf(hexChar, "%02x", mac[runner]);
    strcat(macAddress, hexChar);
    if (runner > 0)
      strcat(macAddress, ":");
  }
}

void display_authCode(StaticJsonDocument<256> hardwareQuery)
{
  if(hardwareQuery["updatedSettings"])
  {
    Serial.println(F("This  Device has not yet been activated."));
    Serial.println(F("Please access the  Device Manager to"));
    Serial.println(F("add this device to your account"));
    Serial.println(hardwareQuery["authCode"].as<const char*>());
  }
};

I'm sure it does, even if I have never used such "delete[]" statement I don't know and I suppose it de-allocates some dynamic memory (allocated with "new" I think)..
I see your "postBody" variable is a string (ok, a "C string") variable you declared as a 256 characters buffer fixed size, inside the "hardware_checkin()" function:

  char postBody[256] = "";

First of all, a "local" variable (e.g. internal to a function) is automatically destroyed on exit from such function (unless declared "static", anyway).
But in this case IMHO it'd be better to define it "global", so the variable "lives" forever in your code, is accessible by any function, you don't need to (can't) "delete" it, avoiding any kind of "allocation/deallocation" issue, and you can always know the amount of allocated memory (at least for large variabiles).
If you have a fixed size string variable and you want to "reset" it, just put a zero as the very first byte:

  //delete[] postBody;  <==-- THIS KILLS THE DEVICE
  postBody[0] = 0;  // This will reset the string buffer to an empty one

It nulls out the first charater , effectively making an "empty string", i.e. lets any string usage completely ignore the following bytes.

If for any reason you also want to make sure the buffer is completely "zeroed", just use "memset" instead (please note the first parameter is the pointer to the buffer, the second one is the byte to write, and the last parameter must represent the full buffer size, e.g. the size you defined on variable declaration, in this case is 256):

  memset(postBody, '\0', 256);

But with a global buffer this is the best solution to get fixed memory allocation and usage flexibility:

...
// Global definitions and variables
#define MAX_BODY 256
char postBody[MAX_BODY] ;
...
  memset(postBody, '\0', MAX_BODY);

HTH

Ah, alright, so better to keep the space allocated and tune the total usage instead of allowing for possible fragmentation with multiple allocations. That's interesting.

Now, if I may...
I'm going to say that your recommendation sounds best to keep that static allocation for sizes that would be small and known (256 bytes or so).
What are your feelings on items that can be more dynamic and larger? For example, the return data from the API call. It could be 256, it could be 2,000 depending on how much there is to display. I don't think that I would want to keep all 3k allocated if it's not used.
Would a quick malloc(length of return API data), convert it to JSON, and then free() be more advantageous in this case?

Although, as I'm thinking though your recommendation and looking at function calls, maybe it would be best to just return the JSON. That would leave that 3k returnData allocated and deleted in the function. However, it would still be dynamically allocated.
That would probably be fine though, right? malloc() for the API response, then consume it and kill that var.

There are a few load -> comp -> update -> process steps in that function where different vars are being used so I'll definitely zero out the ones when I'm done to try to keep things from tripping.

Yes, that's the common usage when dealing with some large and variable size buffers, but it could still have drawbacks (allocating and releasing buffers, with large programs or limited RAM are you sure you'll always have a 3kB contiguous memory available?).
Anyway, any consideration must still be related to the environment in which one operates and the type and frequency of such memory allocation operations, and depends also on the total available memory (e.g. a fixed 3kB buffer could be feasable on a 32kB total RAM board, at least to make things easier to handle).
Said that, I'm sorry but as I don't know neither the MKR features nor the specific data flows/protocols, I think I can't say much more about this topic.

You are being extremely helpful and clearly very knowledgeable. I don't feel that we're talking Arduino here, I think this is a pure best use of memory and I am grateful for your insights.
As I've said, I haven't had to fight direct memory allocation for many years and so have gotten very rusty.

Yes, the 32K is a huge amount, however, with all of the classes loaded (WiFi, JSON, RTC, NTP, Time, etc) I think I'm already down to 18K before I go into this function where the issue is happening.
At least that's what Measuring Memory Usage | Memories of an Arduino | Adafruit Learning System is telling me. This second device has not been "approved" by the application and is only looping in a stage before the bad memory hog and you can see that I'm already down to 18K.

---=== Wake ===---
freeMemory: 19375 B, 18 KB
WiFi join attempt: 1
Attempting to connect to SSID: <redacted>
WiFi status: connected to WiFi
Signal strength in dBm: -33
Connected to SSID: <redacted>
{"success":true,"data":{"approved":false}}
responseBody size: 150
responseJson size: 128
Disconnect/shut down WiFi
---=== Sleep for 5 seconds ===---

That's why I was really hoping for any sort of profiler, so that I can look to see where that initial 14k is being lost, if it really is. Maybe it's not. Do you use any program/function/something to help you manage your UNO's 2K or do you just pen and paper your allocations?

If my max is 3k for that return call string, and json object that pushes me well into 12k right there between the "new" call and the "previous" call for comparison.
That's why I was trying to clear out what I could, when I could. However, that should still leave me 6k to work with, more than you have total... :slight_smile:
This is also the fattest call since it's working with the display API and so it's where I'm fighting the most.
I have seen string that were 2,2xx long unfortunately. At least I'm working with prod data so that I can see the fluctuation now and if I get near 3K from the API, we'll have to shave more from that call because we can't push 4K of data around something like an Ardiuno...

I can say that freeMem call was reporting 3K at first in that fat function, and with adjustments I'm up to 9K but it does not seem to fluctuate within the function when used around malloc() nor free() so there's something else going on in the function that is holding allocation somewhere that the freeMem function cannot account for.

As for the root of the post for anyone else reading this, There seems to have been two core issues.

  1. Memory allocation is still an issue but that is being tightened up. Once there was enough memory to process the return data then I hit a second issue (that might have been caused by memory???)
  2. The WiFi connection (although reported by WiFiNina and connected and ready) was dead causing the wifiClient.connect(domain, 443) command to hang (or crash the Arduino). Specifically, WiFi.status() was equal to WL_CONNECTED, but according to the WAP/firewall/etc, it was not currently connected.

By forcefully disconnecting the WiFi and the end of every loop, that seems to also have stabilized the ability to get past the wifiClient.connect() in any state.

  WiFi.disconnect();
  WiFi.end();

I'm not sure if that is a chip thing or a driver/class thing though. It feels like some overflow where it gets into a bad state.

The second one. I mean, things like using global variables if used by some functions (instead of passing each time their values), global static buffers, and avoiding dynamic allocations where possible (some libraries still allocate memory and/or use "String" variables...). Sometimes I used the same buffer in multiple functions, provided you are sure that won't affect other functions behaviour (e.g. just use it to read a data block/string from a remote source, parse it to get the values, and reset the buffer before exiting the function), so maybe it can't be always applicable. I know in theory it's not exactly a good programming method, but when resources are limited you need to go beyond the obstacle...

For your other questions, I can't be more specific, as I don't have any experience with MKR boards, I hope someone else can help you more than me.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.