($100 paid request) Serial/SafeString/Digital Read/Write bug is ruining my life

No. The == for SafeString just uses SafeStrings and low level c-string methods.
The SafeString definitions do not use Strings at all.

Do they any dynamic memory allocation?

No just static and stack allocations (like char[ ]'s which is actually what are use under the hood).

Edit.
Ignore the rubbish comments below about the SafeString class, cSF( ) etc, doing malloc/free check. They are only used in the SafeString library's SerialComs class.

createSafeString(name,size), or cSF(name,size), is a macro that expands to code like

char name_SAFEBUFFER[size+1];
SafeString name(sizeof(name_SAFEBUFFER),name_SAFEBUFFER, ... );

so no place for memory checks before allocating the char[ ] either statically or on the stack.

However for stack allocations, the SafeString code does check there is sufficient available memory by executing malloc/free, in back to back pairs, to check there is actually free RAM available for the stack to use the will not overwrite the current heap. If the malloc fails, SafeString assumes there will not be enough stack space for the SafeString and sets the capacity to zero which will then provoke SafeString error messages/flags when the SafeString is used.
Once the free memory has been checked (and that memory immediately freed) then the char[ ] is allocated on the stack for use by SafeString internally. No dynamic memory used.

Thinking aloud here - isn't that a risky bet to assume that because malloc() worked then there won't be a stack overflow ? malloc() could have reused a large free block if you had poked holes into the heap and thus succeeded but that does not mean you won't get a stack overflow.

This article offers a way of checking where the Stack Pointer and the Heap Pointer are relative to one another assuming the memory is always organised like this by GCC

His code should be somewhat valid for common AVR based arduino (somewhat because I would argue that having a function call to establish the stack pointer will not be totally precise since calling a function stores stuff on the stack) so a code like this should give a relative OK view

uint8_t * HeapPointer, * StackPointer; // Globally declaring the Stack and Heap pointers.

void setup() {
  Serial.begin(115200); Serial.println();

  HeapPointer = (uint8_t *) malloc(1);    // small allocation.
  free(HeapPointer);                      // release memory
  StackPointer = (uint8_t *)(SP);         // Get the Stack Pointer

  Serial.print(F("StackPointer\t"));
  Serial.println((unsigned int) StackPointer);
  Serial.print(F("HeapPointer\t"));
  Serial.println((unsigned int) HeapPointer);
}

void loop() {}

On other architectures the code would not really work, SP might be undefined, ESP32 uses multiple types of RAM, it also contains multiple heaps with different capabilities and each RTOS task has its own stack allocated from the heap when the task is created)


side question: I would always suggest to use write for sending out single characters with Serial, would that be the same with a BufferedOutput instance and instead of output.print(':'); I would use output. write(':'); ?

Did you consider to reduce the baud rate to prevent overloading the Arduino?

Also I've not seen answers to questions regarding possible challenges with how the gig gets powered. Are you sure everything is fine on that part?

suffers from the 'hole' problem you pointed out.

has more precise values for AVR micros

However for AVR micros (UNO etc) where the memory is really limited, the malloc/free is a resonable test, because if you are not mixing SafeStrings with Strings (i.e not using any dynamic (malloc/free) calls) there will not be any 'holes' in the heap.

For other micros with larger RAM, there is much less likely to be a problem.

both very fair points

Sorry I should have explained I meant my #define s at the beginning which define the message "keys" that the safestring then gets compared to? From what I can tell these are still strings (double quote) and when I change them to single quote for char array I am getting a warning I am having trouble making sense of on the last parameter of the setRelay function

argument of type "int" is incompatible with parameter of type "const char *

very confused why it thinks it is an integer now??

anyhow, I have made a sketch with stream which seems to be producing some kind of failure. It was a little challenging because the spacing between the messages did a decent job at mitigating but some loss is definitely occurring. I still need to attach a secondary serial connection but was curious what your thoughts were.

the message set looks like this

                 "mi1s:1\n"
                 "cainv:1\n"
                 "caexv:1\n"
                 "pump:1\n"
                 "rtinv:1\n"
                 "rtexv:1\n"
                 "rtexv:0\n"
                 "pump:0\n"
                 "mi1s:0\n"
                 "rtinv:0\n"
                 "pump:0\n"
                 "caexv:0\n"
                 "cainv:0\n"

it powers 6 relays and then turns them off, however when I run this I only get replies to the last 7 messages rather than the whole 12. Additionally, there is no issue just turning on the 6 relays simultaneously which should rule out the possibility of a power issue, as well as the fact the relays use low level logic so they are actually using the most of the Arduinos power when all off at the beginning. When I run this sketch with echo on it pulses only the "rtexv" relay very quickly and the messages for all other relays are only 0s.

With this info you think the best next step is to monitor the safestring debug through serial1, try buffering input, or anything else? when I SafeString::setOutput(Serial); on serial0 just to see if there is anything I am not getting any errors etc... just my own SafeString::Output.print

// https://forum.arduino.cc/t/100-paid-request-serial-safestring-digital-read-write-bug-is-ruining-my-life/903396
#include "SafeStringReader.h"
#include "BufferedOutput.h"
#include "SafeStringStream.h"
// serial messages / keys and pin definitions
// air valves
#define CANOPY_IN_AV "cainv"
#define CANOPY_IN_PIN 44 // NC
#define CANOPY_EX_AV "caexv"
#define CANOPY_EX_PIN 47 // OC
#define ROOT_IN_AV "rtinv"
#define ROOT_IN_PIN 43 // NR
#define ROOT_EX_AV "rtexv"
#define ROOT_EX_PIN 46 // OR
#define AMBIENT_IN_AV "aminv"
#define AMBIENT_IN_PIN 45 // NA
#define AMBIENT_EX_AV "amexv"
#define AMBIENT_EX_PIN 48 // OA
// water solenoids
#define PASS_RADIATOR_SOL "pards" // R1
#define PASS_RAD_PIN 39
#define ROOT_RADIATOR_SOL "rtrds" // R2
#define ROOT_RAD_PIN 40
#define CANOPY_SPRAY_SOL "casps" // IC
#define CANOPY_SPRAY_PIN 41
#define CORE_IN_SOL "coins" // IR
#define CORE_IN_PIN 38
#define RETURN_IN_SOL "reins" // RE
#define RETURN_IN_PIN 37
#define ROOT_DRAIN_IN_SOL "rtdrins" // RZ
#define ROOT_DRAIN_IN_PIN 36
#define MIST_1_SOL "mi1s" // M1
#define MIST_1_PIN 35
#define MIST_2_SOL "mi2s" // M2
#define MIST_2_PIN 34
#define DRAIN_SOL "drains" // DR
#define DRAIN_PIN 33
// water pump
#define PUMP "pump"
#define PUMP_PIN 42
// lock
#define LOCK "lock"
#define LOCK_PIN 27
#define EMPTY 26
bool locked = false;
// fan
#define FAN "fan"
#define IN1 5
#define IN2 4
#define PWM 6
// SENSORS
#define HUMIDITY "humid"
#define IN_TEMP "intemp"
#define ROOT_TEMP "rttemp"
#define EX_TEMP "extemp"
#define EX_TEMP_PIN A6
#define PRESSURE "press"
#define PRESSURE_PIN A14
const int pressure_offset = 0.469;
#define RETURN_SOLUTION "resol"
#define RETURN_SOLUTION_PIN 29
#define DOOR_CLOSED "door"
#define LEFT_DOOR_CLOSE_PIN 16
#define RIGHT_DOOR_CLOSE_PIN 15
#define TOUCH "touch"

// SafeStringStream
cSF(sfTestData, 180); // the test data SafeString, will be filled in setup()
cSF(rxBuf, 64); // the extra rxbuffer for the SafeStringStream to mimic the Uno hardware serial rx buffer
SafeStringStream sfStream(sfTestData,rxBuf); // set the SafeString to be read from and the SafeString providing the rx buffer

// create an safeReader instance of SafeStringReader class
// delimited by comma, CarrageReturn or NewLine
// the createSafeStringReader( ) macro creates both the SafeStringReader (safeReader) and the necessary SafeString that holds input chars until a delimiter is found
// args are (ReaderInstanceName, expectedMaxCmdLength, delimiters)
createSafeStringReader(safeReader, 63, ",\r\n");
// Buffered output instance
createBufferedOutput(output, 63, BLOCK_IF_FULL);


bool running = true;
float pressure = 300;
void setup() {
  Serial.begin(115200);

  safeReader.connect(sfStream); // read the test data from the SafeStringStream
  safeReader.echoOn(); // echos the read data back to be re-read again
  sfTestData = F(
                 "mi1s:1\n"
                 "cainv:1\n"
                 "caexv:1\n"
                 "pump:1\n"
                 "rtinv:1\n"
                 "rtexv:1\n"
                 "rtexv:0\n"
                 "pump:0\n"
                 "mi1s:0\n"
                 "rtinv:0\n"
                 "pump:0\n"
                 "caexv:0\n"
                 "cainv:0\n"
               ); // initialized the test data  
  Serial.println("Test Data:-");
  Serial.println(sfTestData);
  Serial.println("----------");
  sfStream.begin(sfTestData, 115200); // start releasing sfTestData at 9600 baud

  pinMode(CANOPY_IN_PIN, OUTPUT);
  pinMode(CANOPY_EX_PIN, OUTPUT);
  pinMode(ROOT_IN_PIN, OUTPUT);
  pinMode(ROOT_EX_PIN, OUTPUT);
  pinMode(AMBIENT_IN_PIN, OUTPUT);
  pinMode(AMBIENT_EX_PIN, OUTPUT);
  pinMode(PASS_RAD_PIN, OUTPUT);
  pinMode(ROOT_RAD_PIN, OUTPUT);
  pinMode(CANOPY_SPRAY_PIN, OUTPUT);
  pinMode(CORE_IN_PIN, OUTPUT);
  pinMode(RETURN_IN_PIN, OUTPUT);
  pinMode(ROOT_DRAIN_IN_PIN, OUTPUT);
  pinMode(MIST_1_PIN, OUTPUT);
  pinMode(MIST_2_PIN, OUTPUT);
  pinMode(DRAIN_PIN, OUTPUT);
  pinMode(PUMP_PIN, OUTPUT);
  pinMode(LOCK_PIN, OUTPUT);
  pinMode(EMPTY, OUTPUT);
  allRelaysOff();
  // enable error messages and SafeString.debug() output to be sent to Serial
  // SafeString::setOutput(Serial);

  // safeReader.connect(Serial); // where SafeStringReader will read from
  // safeReader.echoOn(); // echo back all input, by default echo is off
  output.connect(Serial); // Connect output buffer to serial
}

void loop() {
  output.nextByteOut(); // Send next byte from serial buffer
  processSerial();
}

void processSerial(){
  if(safeReader.read()){
    SafeString::Output.println(safeReader);
    size_t colonIdx = 0;
    colonIdx = safeReader.indexOf(':');
    if (colonIdx < safeReader.length()){ // Command needs data parsing
      // safeString parsing variables
      createSafeString(command, 9); 
      createSafeString(data, 11);
      long dataLong;
      float dataFloat;
      // Parse
      safeReader.substring(command, 0, colonIdx); // extract command before colon
      safeReader.substring(data, colonIdx+1); // extract data after colon
      if (data.toLong(dataLong)){ // type long parse (fails with decimal)
        SafeString::Output.print(F("Long data detected: "));
        SafeString::Output.println(dataLong);
        // output.println(safeReader);
if (command == CANOPY_IN_AV){ // canopy in air valve
          setRelay(CANOPY_IN_PIN, bool(dataLong), CANOPY_IN_AV);
        } else if (command == CANOPY_EX_AV){ // canopy exhaust air valve
          setRelay(CANOPY_EX_PIN, bool(dataLong), CANOPY_EX_AV);
        } else if (command == ROOT_IN_AV){ // root in air valve
          setRelay(ROOT_IN_PIN, bool(dataLong), ROOT_IN_AV);
        } else if (command == ROOT_EX_AV){ // root exhaust air valve
          setRelay(ROOT_EX_PIN, bool(dataLong), ROOT_EX_AV);
        } else if (command == AMBIENT_IN_AV){ // ambient in air valve 
          setRelay(AMBIENT_IN_PIN, bool(dataLong), AMBIENT_IN_AV);
        } else if (command == AMBIENT_EX_AV){ // ambient ehaust air valve
          setRelay(AMBIENT_EX_PIN, bool(dataLong), AMBIENT_EX_AV);
        } else if (command == PASS_RADIATOR_SOL){ // pass through radiator solenoid
          setRelay(PASS_RAD_PIN, bool(dataLong), PASS_RADIATOR_SOL);
        } else if (command == ROOT_RADIATOR_SOL){ // root radiator solenoid
          setRelay(ROOT_RAD_PIN, bool(dataLong), ROOT_RADIATOR_SOL);
        } else if (command == CANOPY_SPRAY_SOL){ // canopy spray solenoid
          setRelay(CANOPY_SPRAY_PIN, bool(dataLong), CANOPY_SPRAY_SOL);
        } else if (command == CORE_IN_SOL){ // core intake solution solenod
          setRelay(CORE_IN_PIN, bool(dataLong), CORE_IN_SOL);
        } else if (command == RETURN_IN_SOL){ // return intake solenoid
          setRelay(RETURN_IN_PIN, bool(dataLong), RETURN_IN_SOL);
        } else if (command == ROOT_DRAIN_IN_SOL){ // root drain intake solenoid
          setRelay(ROOT_DRAIN_IN_PIN, bool(dataLong), ROOT_DRAIN_IN_SOL);
        } else if (command == MIST_1_SOL){ // mist 1 solenoid
          setRelay(MIST_1_PIN, bool(dataLong), MIST_1_SOL);
        } else if (command == MIST_2_SOL){ // mist 2 solenoid
          setRelay(MIST_2_PIN, bool(dataLong), MIST_2_SOL);
        } else if (command == DRAIN_SOL){ // drain solenoid
          setRelay(DRAIN_PIN, bool(dataLong), DRAIN_SOL);
        } else if (command == PUMP){ // water pump
          setRelay(PUMP_PIN, bool(dataLong), PUMP);
        }
      }

    } else { // else no data command
      readPressure();
    }
  }
}

// Send outgoing message with colon
// could pass SafeString & here but seems an overkill
void transmitSerial(const char*key, const char*data) {
  output.print(key); output.print(F(":"));
  output.println(data);
}

// set relay true = on false = off 
void setRelay(int pin, bool setting, const char*reply_key){
  digitalWrite(pin, !setting);
  cSF(sfData, 10);
  sfData = !digitalRead(pin);
  transmitSerial(reply_key, sfData.c_str()); //String(!digitalRead(pin)));
  // transmitSerial(reply_key, String(!setting));
  // Serial.println(reply_key + ':' + String(!digitalRead(pin)));
}

// all relays off
void allRelaysOff(){
  digitalWrite(CANOPY_IN_PIN, HIGH);
  digitalWrite(CANOPY_EX_PIN, HIGH);
  digitalWrite(ROOT_IN_PIN, HIGH);
  digitalWrite(ROOT_EX_PIN, HIGH);
  digitalWrite(AMBIENT_IN_PIN, HIGH);
  digitalWrite(AMBIENT_EX_PIN, HIGH);
  digitalWrite(PASS_RAD_PIN, HIGH);
  digitalWrite(ROOT_RAD_PIN, HIGH);
  digitalWrite(CANOPY_SPRAY_PIN, HIGH);
  digitalWrite(CORE_IN_PIN, HIGH);
  digitalWrite(RETURN_IN_PIN, HIGH);
  digitalWrite(ROOT_DRAIN_IN_PIN, HIGH);
  digitalWrite(MIST_1_PIN, HIGH);
  digitalWrite(MIST_2_PIN, HIGH);
  digitalWrite(DRAIN_PIN, HIGH);
  digitalWrite(PUMP_PIN, HIGH);
  digitalWrite(LOCK_PIN, HIGH);
  digitalWrite(EMPTY, HIGH);
}

// read pressure
void readPressure(){
  static float voltage, pressure;
  // Calculations
  voltage = analogRead(PRESSURE_PIN) * 5.0 / 1024.0; // Calculate the voltage 
  pressure = (voltage - pressure_offset) * 400.0; // Calculate water pressure
  cSF(sfData, 10);
  sfData = pressure;
  transmitSerial(PRESSURE, sfData.c_str());
}


Those defines are c-strings (with a small s) not Strings so no problems there.
Now that your are using SafeStringStream with echo, you have freed up Serial to use for debugging.
Let me play with it here and get back to you.

much appreciated. I did realize this left the serial channel open for debugging but have not been able to get an actual error yet

! added loopTimer and changed output to DROP_IF_FULL which displays ~~ on output buffer overflow.
Still got missing input but no ~~ in output so output buffer not overflowing

10:34:16.856 -> loop uS Latency
10:34:16.856 -> 5sec max:4484 avg:2455
10:34:16.856 -> sofar max:4484 avg:2455 max - prt:2044

latency 2.5ms to 4.4ms
115200 baud => ~87us per byte so ~28bytes arrive per loop
a cmd is ~8bytes => 0.7ms per cmd so ~3.5msgs arrive every loop but only one is processed.

There are at least three solutions
i) change to an ESP32 which processes much faster
ii) reduce the Serial baud rate to say 38400 or 19200. You can use SafeStringStream.RxBufferOverflow() when injecting data to see how many bytes were dropped.
iii) add BufferedInput to cope with bursts of commands.

This code adds the bufferedInput with a 512byte buffer the will hold ~64cmds in a burst
Try that and see how it goes or try one of the other solutions.

// https://forum.arduino.cc/t/100-paid-request-serial-safestring-digital-read-write-bug-is-ruining-my-life/903396
#include "SafeStringReader.h"
#include "BufferedOutput.h"
#include "SafeStringStream.h"
#include "BufferedInput.h"

#include "loopTimer.h"

// serial messages / keys and pin definitions
// air valves
#define CANOPY_IN_AV "cainv"
#define CANOPY_IN_PIN 44 // NC
#define CANOPY_EX_AV "caexv"
#define CANOPY_EX_PIN 47 // OC
#define ROOT_IN_AV "rtinv"
#define ROOT_IN_PIN 43 // NR
#define ROOT_EX_AV "rtexv"
#define ROOT_EX_PIN 46 // OR
#define AMBIENT_IN_AV "aminv"
#define AMBIENT_IN_PIN 45 // NA
#define AMBIENT_EX_AV "amexv"
#define AMBIENT_EX_PIN 48 // OA
// water solenoids
#define PASS_RADIATOR_SOL "pards" // R1
#define PASS_RAD_PIN 39
#define ROOT_RADIATOR_SOL "rtrds" // R2
#define ROOT_RAD_PIN 40
#define CANOPY_SPRAY_SOL "casps" // IC
#define CANOPY_SPRAY_PIN 41
#define CORE_IN_SOL "coins" // IR
#define CORE_IN_PIN 38
#define RETURN_IN_SOL "reins" // RE
#define RETURN_IN_PIN 37
#define ROOT_DRAIN_IN_SOL "rtdrins" // RZ
#define ROOT_DRAIN_IN_PIN 36
#define MIST_1_SOL "mi1s" // M1
#define MIST_1_PIN 35
#define MIST_2_SOL "mi2s" // M2
#define MIST_2_PIN 34
#define DRAIN_SOL "drains" // DR
#define DRAIN_PIN 33
// water pump
#define PUMP "pump"
#define PUMP_PIN 42
// lock
#define LOCK "lock"
#define LOCK_PIN 27
#define EMPTY 26
bool locked = false;
// fan
#define FAN "fan"
#define IN1 5
#define IN2 4
#define PWM 6
// SENSORS
#define HUMIDITY "humid"
#define IN_TEMP "intemp"
#define ROOT_TEMP "rttemp"
#define EX_TEMP "extemp"
#define EX_TEMP_PIN A6
#define PRESSURE "press"
#define PRESSURE_PIN A14
const int pressure_offset = 0.469;
#define RETURN_SOLUTION "resol"
#define RETURN_SOLUTION_PIN 29
#define DOOR_CLOSED "door"
#define LEFT_DOOR_CLOSE_PIN 16
#define RIGHT_DOOR_CLOSE_PIN 15
#define TOUCH "touch"

millisDelay printMissedBytes;
unsigned long PRINT_MISSED_BYTES_MS = 5000;

// SafeStringStream
cSF(sfTestData, 220); // the test data SafeString, will be filled in setup()
cSF(rxBuf, 64); // the extra rxbuffer for the SafeStringStream to mimic the Uno hardware serial rx buffer
SafeStringStream sfStream(sfTestData, rxBuf); // set the SafeString to be read from and the SafeString providing the rx buffer

// create an safeReader instance of SafeStringReader class
// delimited by comma, CarrageReturn or NewLine
// the createSafeStringReader( ) macro creates both the SafeStringReader (safeReader) and the necessary SafeString that holds input chars until a delimiter is found
// args are (ReaderInstanceName, expectedMaxCmdLength, delimiters)
createSafeStringReader(safeReader, 10, ",\r\n"); // max cmd seems to be <= 8 including \n
// Buffered output instance
createBufferedOutput(output, 63, DROP_IF_FULL); //BLOCK_IF_FULL); // will see ~~ in output if this buffer overflows and would block
createBufferedInput(bufferedInput, 512); // handle bursts of input upto (512/8 =) 64cmds

void fillTestData() {
  SafeString::Output.println(">> fillTestData");
  sfTestData = F(
                 "mi1s:1\n"
                 "cainv:1\n"
                 "caexv:1\n"
                 "pump:1\n"
                 "rtinv:1\n"
                 "rtexv:1\n"
                 "rtexv:0\n"
                 "pump:0\n"
                 "mi1s:0\n"
                 "rtinv:0\n"
                 "pump:0\n"
                 "caexv:0\n"
                 "cainv:0\n"
                 "p\n"
                 "mi1s:1\n"
                 "cainv:1\n"
                 "caexv:1\n"
                 "pump:1\n"
                 "rtinv:1\n"
                 "rtexv:1\n"
                 "rtexv:0\n"
                 "pump:0\n"
                 "mi1s:0\n"
                 "rtinv:0\n"
                 "pump:0\n"
                 "caexv:0\n"
                 "cainv:0\n"
                 "p\n"
               ); // initialized the test data
}

bool running = true;
float pressure = 300;
void setup() {
  Serial.begin(115200);
  for (int i = 10; i > 0; i--) {
    Serial.print(i); Serial.print(' '); delay(500);
  }
  Serial.println();
  output.connect(Serial); // Connect output buffer to serial
  // enable error messages and SafeString.debug() output to be sent to Serial
  // SafeString::setOutput(Serial);  // just use SafeString::errorDetected() instead to start with
  bufferedInput.connect(sfStream);
  safeReader.connect(bufferedInput); // read the test data from the SafeStringStream
  safeReader.echoOn(); // echos the read data back to be re-read again
  fillTestData();
  Serial.print("Test Data size:"); Serial.println(sfTestData.length());
  Serial.println("Test Data:-");
  Serial.println(sfTestData);
  Serial.println("----------");
  sfStream.begin(sfTestData, 115200); // start releasing sfTestData at 38400 works 57600 does not nor does 115200);

  pinMode(CANOPY_IN_PIN, OUTPUT);
  pinMode(CANOPY_EX_PIN, OUTPUT);
  pinMode(ROOT_IN_PIN, OUTPUT);
  pinMode(ROOT_EX_PIN, OUTPUT);
  pinMode(AMBIENT_IN_PIN, OUTPUT);
  pinMode(AMBIENT_EX_PIN, OUTPUT);
  pinMode(PASS_RAD_PIN, OUTPUT);
  pinMode(ROOT_RAD_PIN, OUTPUT);
  pinMode(CANOPY_SPRAY_PIN, OUTPUT);
  pinMode(CORE_IN_PIN, OUTPUT);
  pinMode(RETURN_IN_PIN, OUTPUT);
  pinMode(ROOT_DRAIN_IN_PIN, OUTPUT);
  pinMode(MIST_1_PIN, OUTPUT);
  pinMode(MIST_2_PIN, OUTPUT);
  pinMode(DRAIN_PIN, OUTPUT);
  pinMode(PUMP_PIN, OUTPUT);
  pinMode(LOCK_PIN, OUTPUT);
  pinMode(EMPTY, OUTPUT);
  allRelaysOff();

  printMissedBytes.start(PRINT_MISSED_BYTES_MS);
}


void loop() {
  bufferedInput.nextByteIn(); // get next bytes available
  output.nextByteOut(); // Send next byte from serial buffer
  processSerial();
  if (SafeString::errorDetected()) {
    output.println("SafeString Error Detected");
  }
}

void processSerial() {
  if (printMissedBytes.justFinished()) {
    printMissedBytes.restart();
    output.print(">>"); output.println(sfStream.RxBufferOverflow());
  }
  loopTimer.check(output);
  if (safeReader.read()) {
    //  SafeString::Output.print(">");
    //  SafeString::Output.println(safeReader);
    size_t colonIdx = 0;
    colonIdx = safeReader.indexOf(':');
    if (colonIdx < safeReader.length()) { // Command needs data parsing
      // safeString parsing variables
      createSafeString(command, 9);
      createSafeString(data, 11);
      long dataLong;
      // Parse
      safeReader.substring(command, 0, colonIdx); // extract command before colon
      safeReader.substring(data, colonIdx + 1); // extract data after colon
      if (data.toLong(dataLong)) { // type long parse (fails with decimal)
        // SafeString::Output.print(F("Long data detected: "));
        // SafeString::Output.println(dataLong);
        // output.println(safeReader);
        if (command == CANOPY_IN_AV) { // canopy in air valve
          setRelay(CANOPY_IN_PIN, dataLong, CANOPY_IN_AV);
        } else if (command == CANOPY_EX_AV) { // canopy exhaust air valve
          setRelay(CANOPY_EX_PIN, dataLong, CANOPY_EX_AV);
        } else if (command == ROOT_IN_AV) { // root in air valve
          setRelay(ROOT_IN_PIN, dataLong, ROOT_IN_AV);
        } else if (command == ROOT_EX_AV) { // root exhaust air valve
          setRelay(ROOT_EX_PIN, dataLong, ROOT_EX_AV);
        } else if (command == AMBIENT_IN_AV) { // ambient in air valve
          setRelay(AMBIENT_IN_PIN, dataLong, AMBIENT_IN_AV);
        } else if (command == AMBIENT_EX_AV) { // ambient ehaust air valve
          setRelay(AMBIENT_EX_PIN, dataLong, AMBIENT_EX_AV);
        } else if (command == PASS_RADIATOR_SOL) { // pass through radiator solenoid
          setRelay(PASS_RAD_PIN, dataLong, PASS_RADIATOR_SOL);
        } else if (command == ROOT_RADIATOR_SOL) { // root radiator solenoid
          setRelay(ROOT_RAD_PIN, dataLong, ROOT_RADIATOR_SOL);
        } else if (command == CANOPY_SPRAY_SOL) { // canopy spray solenoid
          setRelay(CANOPY_SPRAY_PIN, dataLong, CANOPY_SPRAY_SOL);
        } else if (command == CORE_IN_SOL) { // core intake solution solenod
          setRelay(CORE_IN_PIN, dataLong, CORE_IN_SOL);
        } else if (command == RETURN_IN_SOL) { // return intake solenoid
          setRelay(RETURN_IN_PIN, dataLong, RETURN_IN_SOL);
        } else if (command == ROOT_DRAIN_IN_SOL) { // root drain intake solenoid
          setRelay(ROOT_DRAIN_IN_PIN, dataLong, ROOT_DRAIN_IN_SOL);
        } else if (command == MIST_1_SOL) { // mist 1 solenoid
          setRelay(MIST_1_PIN, dataLong, MIST_1_SOL);
        } else if (command == MIST_2_SOL) { // mist 2 solenoid
          setRelay(MIST_2_PIN, dataLong, MIST_2_SOL);
        } else if (command == DRAIN_SOL) { // drain solenoid
          setRelay(DRAIN_PIN, dataLong, DRAIN_SOL);
        } else if (command == PUMP) { // water pump
          setRelay(PUMP_PIN, dataLong, PUMP);
        } else {
        }
      } else {
      }
    } else { // else no data command
      readPressure();
    }
  }
}

// Send outgoing message with colon
// could pass SafeString & here but seems an overkill
void transmitSerial(const char*key, const char*data) {
  output.print(key); output.print(F(":"));
  output.println(data);
}

// set relay true = on false = off
void setRelay(int pin, bool setting, const char*reply_key) {
  digitalWrite(pin, !setting);
  cSF(sfData, 10);
  sfData = setting; //!digitalRead(pin); // note this does not work on ESP32 use setting instead
  transmitSerial(reply_key, sfData.c_str()); //String(!digitalRead(pin)));
}

// all relays off
void allRelaysOff() {
  digitalWrite(CANOPY_IN_PIN, HIGH);
  digitalWrite(CANOPY_EX_PIN, HIGH);
  digitalWrite(ROOT_IN_PIN, HIGH);
  digitalWrite(ROOT_EX_PIN, HIGH);
  digitalWrite(AMBIENT_IN_PIN, HIGH);
  digitalWrite(AMBIENT_EX_PIN, HIGH);
  digitalWrite(PASS_RAD_PIN, HIGH);
  digitalWrite(ROOT_RAD_PIN, HIGH);
  digitalWrite(CANOPY_SPRAY_PIN, HIGH);
  digitalWrite(CORE_IN_PIN, HIGH);
  digitalWrite(RETURN_IN_PIN, HIGH);
  digitalWrite(ROOT_DRAIN_IN_PIN, HIGH);
  digitalWrite(MIST_1_PIN, HIGH);
  digitalWrite(MIST_2_PIN, HIGH);
  digitalWrite(DRAIN_PIN, HIGH);
  digitalWrite(PUMP_PIN, HIGH);
  digitalWrite(LOCK_PIN, HIGH);
  digitalWrite(EMPTY, HIGH);
}

// read pressure
void readPressure() {
  static float voltage, pressure;
  // Calculations
  voltage = analogRead(PRESSURE_PIN) * 5.0 / 1024.0; // Calculate the voltage
  pressure = (voltage - pressure_offset) * 400.0; // Calculate water pressure
  cSF(sfData, 10);
  sfData = pressure;
  transmitSerial(PRESSURE, sfData.c_str());
}

thank you so much you're seriously saving me. Initial testing seems to be working, I would like to test a bit more in application but I think this will work!

send me a PM and we can arrange payment

I suggest that you remove your email address unless you want to receive spam. You can message most users here on the forum.

thanks I did not see the option so I assumed there wasn't one. maybe its just because of my new acc or something

Actually they are neither.. The compiler just replaces the keyword in the code with whatever is behind it. Literals is what they are.

Since the current fix only papers over handling bursts of commands, I suggest you add, on the Python side, a list of command sent (containing : ) and then remove them as transmitSerial sends back the results.
Raise an error if the next response from Arduino, with a :, does not match the top of the list.

Then reduce the size of bufferedInput until you provoke an input overflow error and see that your checking code detects it.

So currently it appears the input buffer is successfully stopping that message loss (at least with the stream) but when I make some changes and try to use it in practice I am getting the same failure. I am doing a few things to try and determine if the overflow is actually occurring during run (setting up serial1 monitoring and making it so host can check the overflow by request). I will let you know what I find but I was curious if there is a technical purpose for the 5 sec delay at the beginning?

Reducing the buffer to create and detect an overflow also makes sense, I will get a python only script made to produce the bug but I do not quite understand the purpose of tracking all the sent messages since I have no problem seeing which one fails on that side?

I may have made some mistakes when trying to set it to take info from serial instead of the stream. Is this correct?

 safeReader.connect(bufferedInput);

also does this check for overflow of the input buffer or the set stream?

sfStream.RxBufferOverflow()

Looks like you need to debug live via Serial1 now.

The 5 sec delay is not necessary for Mega which restarts when you open the IDE monitor, but some other boards (on my pc) don't like the monitor being open while programming and restart as soon as the upload finishes so I have a habit of putting this in so I can see I have not missed any of the startup msgs. I often swap sketches from board to board for testing.

No need to add the checks, if that is the case.

Yes that is correct.

RxBufferOverflow() only checks overflows for the sfStream :frowning: so no use once you move to live testing.

thanks for the clarification :slightly_smiling_face: moving to serial1 debugging now