ESP32 prints out extra character from Arduino serial data

Hey guys, I hope I’m in the right place.

I have an ESP32 that receives serial data from an Arduino nano.

The project of which this is a part, is a plant watering system with multiple plants(7 at the moment).

The ESP32 code to receive the data, prints out the data string(an influxdb protocol string) and then sends it to a broker via MQTT.

void readSoftwareSerial2() {

  if (Serial2.available()) {

    noInterrupts();
    char output[255];
    Serial2.readBytesUntil('\r', output, 255);
    // Flush the serial
    while (Serial2.read() >= 0)
    Serial.println(output);
    client.publish(MQTT_SERIAL_PUBLISH_PLANTS, output);
    delay(10);
    interrupts();
  }
}

The full string that is sent for each plant should look something like this.
"plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=various,plant_id=4 servo_pin=5,read_pin=17,sensor_reading=435,moi_setting_high=90,moi_setting_low=60,sensor_low_value=339,moisture_level=70,in_dead_zone=t,servo_open=f,pump_open=f
"

The problem is that for ONE plant only(plant id 6) the last value is always incorrect. An extra character if added. Instead of “pump_open=f” it always reads “pump_open=ff”.

But the serial information sent from the nano is as it should be:

The code from the nano is like so

for (int i = 0; i < MODULE_COUNT; i++)
    {
        Module currentModule = modules[i];
        int totalChar = strlen(systemId);
        // Project /
        Serial.print("plant_system");
        //Tags /
        Serial.print(",city=Berlin");
        Serial.print(",location=oderstrasse");
        Serial.print(",room=andrews");
        Serial.print(",plant_type=");
        Serial.print(currentModule.plantType);
        Serial.print(",plant_id=");
        Serial.print(currentModule.id);
        
        Serial.print(" ");

        //Fields
        Serial.print("servo_pin=");
        Serial.print(currentModule.servoPin);

        Serial.print(",read_pin=");
        Serial.print(currentModule.readPin);

        Serial.print(",sensor_reading=");
        Serial.print(analogRead(currentModule.readPin));

        Serial.print(",moi_setting_high=");
        Serial.print(currentModule.moistureSettingHigh);

        Serial.print(",moi_setting_low=");
        Serial.print(currentModule.moistureSettingLow);

        Serial.print(",sensor_low_value=");
        Serial.print(currentModule.sensorUpperValue);

        currentModule.currentPercentage = convertToPercent(analogRead(currentModule.readPin), currentModule);

        Serial.print(",moisture_level=");
        Serial.print(currentModule.currentPercentage);
        
        if (currentModule.currentPercentage < currentModule.moistureSettingLow)
        {
            currentModule.isPumping = true;
            needsPump = true;
            digitalWrite(currentModule.servoPin, LOW);

            //Opening servo
            Serial.print(",in_dead_zone=f");
        }
        if (currentModule.currentPercentage >= currentModule.moistureSettingLow && currentModule.currentPercentage <= currentModule.moistureSettingHigh)
        {
            if (!currentModule.isPumping)
            {
                digitalWrite(currentModule.servoPin, HIGH);
            }

            //the deadzone
            Serial.print(",in_dead_zone=t");
        }
        if (currentModule.currentPercentage > currentModule.moistureSettingHigh)
        {
            currentModule.isPumping = false;
            digitalWrite(currentModule.servoPin, HIGH);

            //Opening servo
            Serial.print(",in_dead_zone=f");
        }

        byte servoPinState = digitalRead(currentModule.servoPin);
        if (servoPinState == LOW)
        {
            Serial.print(",servo_open=t");
        }
        else
        {
            Serial.print(",servo_open=f");
        } 

        byte pumpPinState = digitalRead(pumpPin);
        if (pumpPinState == LOW)
        {
            Serial.println(",pump_open=f");
        }
        else
        {
            Serial.println(",pump_open=t");
        }
        delay(10);
    }

So I ask has anyone ever come across this kind of behavior before,

AND

Does anyone have any idea on why it happens, and how to fix it?

Any help would be much appreciated.

Cheers

Andrew

noInterrupts(); Why?

I read that it helps... and it turns out it does.

To be honest I don't fully understand why.

If you look that the image below this is the output of the receiving ESP32 even more incorrect characters are added.

|500x305

earyzhe: Hey guys, I hope I'm in the right place.

But the serial information sent from the nano is as it should be: |500x83

The info as printed by the Nano is not as it should be: If you look at the Nano printout, you will see that the printout for line 6 is not properly printing. If the ESP32 receives garbage, the ESP32 will process garbage. The ESP32 is doing what it has been coded to do.

The print out from the serial of the Nano is fine and how it should be.... unless I'm misunderstanding something.

See the image below which shows cycling through everything including line 6.

Why is line 6 indented? Why does line 7 start with a "/o_"?

Would it not stand to reason that something happens with or after line 5 that pushes and error downstream? The ESP32 is not at fault for processing garbage out with garbage in.

6 is not indented. It's just that some of the info further up the line is less than for that line.

In this case, 7 had a key-value pair of "plant_type=peace_lily" and 6 "plant_type=bonsai", giving the look of indentation.

And there was no "/o" . It was the last half of a "v" e.g , "servo_open=f,pump_open=f"

To be clear, the print out for the entire 7 plants straight from the nano is ok, and appears like this in the serial monitor.

13:59:13.232 -> plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=unknown,plant_id=1 servo_pin=2,read_pin=14,sensor_reading=360,moi_setting_high=70,moi_setting_low=40,sensor_low_value=323,moisture_level=88,in_dead_zone=f,servo_open=f,pump_open=f
13:59:13.266 -> plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=unknown,plant_id=2 servo_pin=3,read_pin=15,sensor_reading=336,moi_setting_high=70,moi_setting_low=40,sensor_low_value=323,moisture_level=95,in_dead_zone=f,servo_open=f,pump_open=f
13:59:13.333 -> plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=unknown,plant_id=3 servo_pin=4,read_pin=16,sensor_reading=319,moi_setting_high=70,moi_setting_low=40,sensor_low_value=318,moisture_level=99,in_dead_zone=f,servo_open=f,pump_open=f
13:59:13.367 -> plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=various,plant_id=4 servo_pin=5,read_pin=17,sensor_reading=435,moi_setting_high=90,moi_setting_low=60,sensor_low_value=339,moisture_level=70,in_dead_zone=t,servo_open=f,pump_open=f
13:59:13.400 -> plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=bonsai,plant_id=5 servo_pin=6,read_pin=18,sensor_reading=313,moi_setting_high=70,moi_setting_low=40,sensor_low_value=308,moisture_level=98,in_dead_zone=f,servo_open=f,pump_open=f
13:59:13.469 -> plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=cactus,plant_id=6 servo_pin=7,read_pin=19,sensor_reading=471,moi_setting_high=60,moi_setting_low=30,sensor_low_value=372,moisture_level=69,in_dead_zone=f,servo_open=f,pump_open=f
13:59:13.502 -> plant_system,city=Berlin,location=oderstrasse,room=andrews,plant_type=peace_lily,plant_id=7 servo_pin=8,read_pin=20,sensor_reading=314,moi_setting_high=70,moi_setting_low=40,sensor_low_value=287,moisture_level=90,in_dead_zone=f,servo_open=f,pump_open=f

I am sticking with garbage in garbage out.

OK so you think there is garbage being passed into the serial which is received by the ESP32, but the serial monitor is not picking it up on the nano side? Is there a way to view what that may be? and any suggestions on how to get rid of it based on the code provided? How would this garbage be created in the first place?

It comes at the end of each line but only line 6(90% of the time). I cannot see I am doing anything different with that line as it is just in the for loop, going through the same code as every other one.

Could it be something to do with "println"? I'm using that so I can use "readBytesUntil" on the other side. But could this be something todo with it?

On the ESP32, I’d use freeRTOS.

On the ESP32 you have 3 serial ports, why use slow software serial when, faster, hardware serial would work much better?

I’d comment out all the rest of the serial print statements, Nano, and just let the error portion print to see if that is the source of the error.

I’d only have one Serial.println(); just before the delay.

I’d send the data as sentences with begin sentence and end sentence markers like so:

void fSendSerialToBrain( void *pvParameters )
{
  struct stu_LIDAR_INFO pxLIDAR_INFO;
  xSemaphoreGive ( sema_SendSerialToBrain );
  String sSerial;
  sSerial.reserve ( 300 );
  for (;;)
  {
    xEventGroupWaitBits (eg, evtSendSerialToBrain, pdTRUE, pdTRUE, portMAX_DELAY);
    if ( xSemaphoreTake( sema_SendSerialToBrain, xZeroTicksToWait ) == pdTRUE ) // grab semaphore, no wait
    {
      int CellCount = 1;
      xSemaphoreTake( sema_LIDAR_INFO, xSemaphoreTicksToWait );
      xQueueReceive ( xQ_LIDAR_INFO, &pxLIDAR_INFO, QueueReceiveDelayTime );
      xSemaphoreGive( sema_LIDAR_INFO );
      sSerial.concat( "<!," ); //sentence begin
      sSerial.concat( String(ScanPoints) + "," );
      for ( CellCount; CellCount <= ScanPoints; CellCount++ )
      {
        sSerial.concat( String(pxLIDAR_INFO.Range[CellCount]) + "," );
      }
      sSerial.concat( ">" ); //sentence end
      SerialBrain.println ( sSerial );
      sSerial = "";
    }
  }
  vTaskDelete( NULL );
} // void fSendSerialToBrain( void *pvParameters )

That way the receiving end would only process compelte sentences instead partial sentences.
I’d only process, on the receiving end, complete sentences and ignore non complete sentences like so:

void fReceiveSerial_LIDAR( void * parameters  )
{
  bool BeginSentence = false;
  sSerial.reserve ( StringBufferSize300 );
  char OneChar;
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtReceiveSerial_LIDAR, pdTRUE, pdTRUE, portMAX_DELAY);
    if ( LIDARSerial.available() >= 1 )
    {
      while ( LIDARSerial.available() )
      {
        OneChar = LIDARSerial.read();
        if ( BeginSentence )
        {
          if ( OneChar == '>')
          {
            if ( xSemaphoreTake( sema_ParseLIDAR_ReceivedSerial, xSemaphoreTicksToWait10 ) == pdTRUE )
            {
              xQueueOverwrite( xQ_LIDAR_Display_INFO, ( void * ) &sSerial );
              xEventGroupSetBits( eg, evtParseLIDAR_ReceivedSerial );
            }
            BeginSentence = false;
            break;
          }
          sSerial.concat ( OneChar );
        }
        else
        {
          if ( OneChar == '<' )
          {
            sSerial = ""; // clear string buffer
            BeginSentence = true; // found begining of sentence
          }
        }
      } //  while ( LIDARSerial.available() )
    } //if ( LIDARSerial.available() >= 1 )
    xSemaphoreGive( sema_ReceiveSerial_LIDAR );
  }
  vTaskDelete( NULL );
} //void fParseSerial( void * parameters  )

I’d use code to parse the received serial, with the ESP32, I’d put the parsing code on the other core.
Parse example:

void fParseLIDAR_ReceivedSerial ( void * parameters )
{
  // distribute received LIDAR info
  String sTmp = "";
  sTmp.reserve ( 20 );
  String sMessage = "";
  sMessage.reserve ( StringBufferSize300 );
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtParseLIDAR_ReceivedSerial, pdTRUE, pdTRUE, portMAX_DELAY) ;
    xQueueReceive ( xQ_LIDAR_Display_INFO, &sMessage, QueueReceiveDelayTime );
    int commaIndex = sMessage.indexOf(',');
    sTmp.concat ( sMessage.substring(0, commaIndex) );
    sMessage.remove( 0, (commaIndex + 1) ); // chop off begining of message
    if ( sTmp == "!" )
    {
      xSemaphoreGive ( sema_LIDAR_OK );
      //  Display info from LIDAR
      sLIDAR_Display_Info = sMessage;
    }
    if ( sTmp == "$" )
    {
      xEventGroupSetBits( eg1, evtResetWatchDogVariables );
    }
    if ( sTmp == "#")
    {
      xSemaphoreTake( sema_LIDAR_Alarm, xSemaphoreTicksToWait );
      sLIDAR_Alarm_info = sMessage;
      xSemaphoreGive( sema_LIDAR_Alarm );
      xEventGroupSetBits( eg, evtfLIDAR_Alarm );
    }
    sTmp = "";
    sMessage = "";
    xSemaphoreGive( sema_ParseLIDAR_ReceivedSerial );
  }
  vTaskDelete( NULL );
} // void fParseReceivedSerial ( void * parameters )

Notice the use of if ( LIDARSerial.available() >= 1 ) I found that the DUE, STM32, and ESP32 work better with if ( LIDARSerial.available() >= 1 ) instead of if ( LIDARSerial.available() >= 0 ).

In other words, I’d add some formality to the send and receive of Serial data sent between devices.

Yes, I agree complete sentences would be a good idea. I was doing it that way originally. I created a string and concatenated the information until it was all there and then I sent it to the serial port.

When I did it this way though, my program would only work for 20 minutes or so because I think the string class was causing memory fragmenting issues.

I chose to do it this way to stop that, and it solved that problem but created another.

I did now do something similar to what you said about putting markers on the start and the end of the message. Basically, I followed this guide: https://forum.arduino.cc/index.php?topic=288234.0

I was trying to resist and do it the easy way but of course, it didn't work because Arduino. >:(

Seems to be working now consistently so thank for your help and sticking with me :)

Andrew

earyzhe: Yes, I agree complete sentences would be a good idea. I was doing it that way originally. I created a string and concatenated the information until it was all there and then I sent it to the serial port.

When I did it this way though, my program would only work for 20 minutes or so because I think the string class was causing memory fragmenting issues.

I chose to do it this way to stop that, and it solved that problem but created another.

I did now do something similar to what you said about putting markers on the start and the end of the message. Basically, I followed this guide: https://forum.arduino.cc/index.php?topic=288234.0

I was trying to resist and do it the easy way but of course, it didn't work because Arduino. >:(

Seems to be working now consistently so thank for your help and sticking with me :)

Andrew

If, when using strings you would create a String buffer to do the string work in, you would not get memory fragmentation issues.

I made changes to my previous post that you might want to read over.

Cool. Good to know. Thanks :)

You are seeing leftovers from previous readBytesUntil() calls, because does not add a null terminator to the buffer. You have to add it yourself:

size_t len = Serial2.readBytesUntil('\r', output, sizeof(output)-1);
output[len] = 0;

Ahh ok good to know thanks :). But still, find it weird how it was only happening for that one line all the time. Even though it was the same as the others. 6/7 lines it worked fine....