How to handle json parse errors ?

Projects Programming
1

Hi everyone,

I am working on a project, where I have an Arduino and an ESP32 exchanging data via Serial communication and Json.
Everything works well, except at some point, where the Json can not be parsed by the Arduino, and the Arduino seems to not react to any message from the ESP32 anymore..

The code of this part is :

  while(Serial.available()) { // When a message becomes available, run the while and reads the message
    message = Serial.readString();
    messageReady = true;
  }
  
  // Only process message if there's one
  if(messageReady) {
    DynamicJsonBuffer jb; // Create a Json Buffer for message storage
     JsonObject& obj = jb.parseObject(message); // Create a Json out of the message received

if (!obj.success()) {
      //jb.clear();
        //  messageReady=false;
    //  goto tryagain;
     // Serial.println("parseObject() failed"); // In case the message is not working ( Happens usualy when the ESP is reset at the first loop )
     void(* resetFunc) (void) = 0;//declare reset function at address 0
     resetFunc(); //call reset
}

The problem comes when due to « ParseObject failed ». Missing the message is not a problem, as the ESP32 will send the same one some second after because it did not receive an answer. The biggest problem is that, even if the ESP32 sends another message, the Arduino is still « stuck »due to the first error, and the upcoming message are still « parse failed ».
The only way I found around it is to reset the arduino, (by code, like you can see in the commented region, or via the button).
But this methods arise other problems.

I tried so many things to « reset » the message, but I couldn’t . I tried things like « jb.clear » and then «goto » earlier in the code to wait for another message, but that did not work.
It seems I have to find a way to empty, or reset the current message… Not sure.
Do you have an idea how to manage this type of error ? Thanks !

2

Perhaps your JSON strings are longer than the input buffer size and aren't being read fast enough. What is the nominal size of the JSON string, what is the frequency the JSON is transmitted, and how often are you parsing the JSON?

Can you post your full code for both modules?

3

The JSON strings size are not a problem because they work in nominal mode. They are like 20/30 characters.
It is just that once an error occurs in the parsing, all the upcoming messages will not be read.
The JSON frequency is low, because it happens when I choose, so basically it is like this :
I tell my ESP32 to send a request message ==> The ESP32 sends a JSON with a header "request" ==> The Arduino receive the request, takes some measurements, and then send another JSON with the header "Answer" ==> done.

It works quite well, but if somehow a problem arise and the error with the json arriving, it will not work anymore and even though it received the upcoming message, will not be able to parse.
Since it work if I reboot it, the problem is not in the message, but in the treatment of it by the Arduino.

That is why I would like to know how I can "reset and empty" the communication, without reseting the whole arduino.

I will send a "resumed" version of the code

ESP32 :

void loop()
{
   tryagain:// Used to send another request if no answer comes

  Firebase.getInt(firebaseData, "/Request/Requestall");
  int requestdata=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/UpdateLight");
  int requestupdatelight=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/PumpChange");
  int requestpump=firebaseData.intData();

  while(requestdata == 0 && requestupdatelight==0 && requestpump==0) {// Block while there is no request from user
    
  delay(1000);

  Firebase.getInt(firebaseData, "/Request/Requestall");
  requestdata=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/UpdateLight");
  requestupdatelight=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/PumpChange");
  requestpump=firebaseData.intData();

    
    }

 if(requestdata>0){
   req["type"] = "request";

   req.printTo(Serial); 

 boolean messageReady = false;
 String message = "";
 int i=0;
 
  while(messageReady == false) { // Runs until a message is received back
    
      if(Serial.available()) {
       message = Serial.readString(); // Read the message
       messageReady = true;  // "Break" the while and continue
       continue; //skip the delay if necessary
     }
     
     delay(10);
     i++;
     if(i % 1000 == 0) {goto tryagain;}// If after 10s nothing comes back, tries again to send a request
  }

    JsonObject& obj = jb.parseObject(message); // Translate the received message into a JsonObject for treatment
    lux = obj["lux"];
    dist= obj["dist"];
    temp= obj["temp"];
    humidity= obj["humidity"];
    waterlevel= obj["waterlevel"];
    tdsValue=obj["tdsvalue"];
    watertemp=obj["watertemp"];

    obj.prettyPrintTo(Serial);
    Serial.println();

    Firebase.setDouble(firebaseData,"Sensors/Light", lux);
    Firebase.setDouble(firebaseData,"Sensors/TankHeight", dist);
    Firebase.setDouble(firebaseData,"Sensors/AirTemp", temp);
    Firebase.setDouble(firebaseData,"Sensors/Humidity", humidity);
    Firebase.setDouble(firebaseData,"Sensors/WaterLevel", waterlevel);
    Firebase.setDouble(firebaseData,"Sensors/EC", tdsValue);
    Firebase.setDouble(firebaseData,"Sensors/WaterTemp", watertemp);


    
    Firebase.setDouble(firebaseData,"Request/Requestall", 0); // Reset the request
    requestupdatelight=0;
 }

For the arduino :

 tryagain:// If message bugs, retries
  // measurements
    uint16_t lux = lightMeter.readLightLevel();
    tdsValue=getECValue();
    int temp = bme.readTemperature();
    int humidity = bme.readHumidity();
    waterlevel = analogRead(respin);
    dist=DistanceCalculation();
    watertemp=sensors.getTempCByIndex(0);

    
  // Monitor serial communication
  while(Serial.available()) { // When a message becomes available, run the while and reads the message
    message = Serial.readString();
    messageReady = true;
  }
  
  // Only process message if there's one
  if(messageReady) {
    DynamicJsonBuffer jb; // Create a Json Buffer for message storage
     JsonObject& obj = jb.parseObject(message); // Create a Json out of the message received

if (!obj.success()) {
      //jb.clear();
        //  messageReady=false;
    //  goto tryagain;
     // Serial.println("parseObject() failed"); // In case the message is not working ( Happens usualy when the ESP is reset at the first loop )
     void(* resetFunc) (void) = 0;//declare reset function at address 0
     resetFunc(); //call reset
}


    if(obj["type"] == "request") { // If it is a request
      obj["type"] = "response"; // The sent back message will be a response
      obj["lux"] = lux; // Write the lux value in the message
      obj["dist"]=dist;
      obj["temp"]=temp;
      obj["humidity"]=humidity;
      obj["waterlevel"]=waterlevel;
      obj["tdsvalue"]=tdsValue;
      obj["watertemp"]=watertemp;

      obj.printTo(Serial); // Send the message

    }
4

Please post the complete programs for each module.

5

Arduino

/*------------------------------------------------------------------------------
 
------------------------------------------------------------------------------*/
#include <ArduinoJson.h>
#include <BH1750.h>
#include <Wire.h>
#include <SPI.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h>
#include "RBDdimmer.h"
#include <OneWire.h>
#include <DallasTemperature.h>



// Sensors 
BH1750 lightMeter;
Adafruit_BME280 bme; // I2C
int respin = A2; // sensor pin used for WATER LEVEL
const int trigPin = 9;
const int echoPin = 10;
#define TdsSensorPin A1
#define outputPin  3 
#define zerocross  2 // for boards with CHANGEBLE input pins
#define ONE_WIRE_BUS 5
int relayPump = 7;



dimmerLamp dimmer(outputPin); //initialase port for dimmer for MEGA, Leonardo, UNO, Arduino M0, Arduino Zero



// Values
#define SEALEVELPRESSURE_HPA (1013.25)
#define VREF 5 // analog reference voltage(Volt) of the ADC
#define SCOUNT 30 // sum of sample point


// Variables
String message = "";
bool messageReady = false;
int waterlevel = 0;  // holds the value
long duration;
int dist;
int analogBuffer[SCOUNT]; // store the analog value in the array, read from ADC
int analogBufferTemp[SCOUNT];
int analogBufferIndex = 0,copyIndex = 0;
float averageVoltage = 0,tdsValue = 0,temperature = 25; // METTRE LA VALEUR DU CAPTEUR DE TEMPERATURE D'EAU
int lightdim=0;
OneWire oneWire(ONE_WIRE_BUS);  
DallasTemperature sensors(&oneWire);
int watertemp;
int pumpstate;
volatile int flow_frequency; // Measures flow sensor pulses
unsigned int l_hour; // Calculated litres/hour
unsigned long currentTime;
unsigned long cloopTime;



void setup() {
    Serial.begin(115200);

  pinMode(trigPin, OUTPUT); // Sets the trigPin as an Output
  pinMode(echoPin, INPUT); // Sets the echoPin as an Input
  pinMode(TdsSensorPin,INPUT);
  dimmer.begin(NORMAL_MODE, ON); //dimmer initialisation: name.begin(MODE, STATE) 
  sensors.begin();
  sensors.requestTemperatures(); 
  pinMode(relayPump, OUTPUT);
  digitalWrite(relayPump, LOW);
  bme.begin(0x76);
    
     lightMeter.begin();


}



void loop() {
    tryagain:// If message bugs, retries
  // measurements
    uint16_t lux = lightMeter.readLightLevel();
    tdsValue=getECValue(); // Ec Value is the old value for some reasons ?
    int temp = bme.readTemperature();
    int humidity = bme.readHumidity();
    waterlevel = analogRead(respin);
    dist=DistanceCalculation();
    watertemp=sensors.getTempCByIndex(0);

    
  // Monitor serial communication
  while(Serial.available()) { // When a message becomes available, run the while and reads the message
    message = Serial.readString();
    messageReady = true;
  }
  
  // Only process message if there's one
  if(messageReady) {
    DynamicJsonBuffer jb; // Create a Json Buffer for message storage
     JsonObject& obj = jb.parseObject(message); // Create a Json out of the message received

if (!obj.success()) {
      //jb.clear();
        //  messageReady=false;
    //  goto tryagain;
     // Serial.println("parseObject() failed"); // In case the message is not working ( Happens usualy when the ESP is reset at the first loop )
     void(* resetFunc) (void) = 0;//declare reset function at address 0
     resetFunc(); //call reset
}


    if(obj["type"] == "request") { // If it is a request
      obj["type"] = "response"; // The sent back message will be a response
      // Get data from analog sensors
      obj["lux"] = lux; // Write the lux value in the message
      obj["dist"]=dist;
      obj["temp"]=temp;
      obj["humidity"]=humidity;
      obj["waterlevel"]=waterlevel;
      obj["tdsvalue"]=tdsValue;
      obj["watertemp"]=watertemp;
      lightdim=obj["lightdim"];

      obj.printTo(Serial); // Send the message
      dimmer.setPower(lightdim);

    }

    if(obj["type"] == "control") {
      
    lightdim=obj["lightdim"];
    dimmer.setPower(lightdim); // setPower(0-100%);

    pumpstate=obj["pumpstate"];

    if(pumpstate==1){
          digitalWrite(relayPump, HIGH);
    }
    else{
          digitalWrite(relayPump, LOW);
    }
      
    }
    
    messageReady = false;
  }

  
}

int DistanceCalculation(){
  int distance;
  digitalWrite(trigPin, LOW);
  delayMicroseconds(2);
// Sets the trigPin on HIGH state for 10 micro seconds
  digitalWrite(trigPin, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin, LOW);
// Reads the echoPin, returns the sound wave travel time in microseconds
  duration = pulseIn(echoPin, HIGH);
// Calculating the distance
  distance= duration*0.034/2;

  return distance;
}



int getMedianNum(int bArray[], int iFilterLen){
int bTab[iFilterLen];
for (byte i = 0; i<iFilterLen; i++)
bTab[i] = bArray[i];
int i, j, bTemp;
for (j = 0; j < iFilterLen - 1; j++)
{
for (i = 0; i < iFilterLen - j - 1; i++)
{
if (bTab[i] > bTab[i + 1])
{
bTemp = bTab[i];
bTab[i] = bTab[i + 1];
bTab[i + 1] = bTemp;
}
}
}
if ((iFilterLen & 1) > 0)
bTemp = bTab[(iFilterLen - 1) / 2];
else
bTemp = (bTab[iFilterLen / 2] + bTab[iFilterLen / 2 - 1]) / 2;
return bTemp;
}


int getECValue(){
    static unsigned long analogSampleTimepoint = millis();
  
  if(millis()-analogSampleTimepoint > 40U) //every 40 milliseconds,read the analog value from the ADC
{
    analogSampleTimepoint = millis();
    analogBuffer[analogBufferIndex] = analogRead(TdsSensorPin); //read the analog value and store into the buffer
    analogBufferIndex++;
    
  if(analogBufferIndex == SCOUNT)
  analogBufferIndex = 0;
}
  static unsigned long printTimepoint = millis();
  if(millis()-printTimepoint > 800U)
{
    printTimepoint = millis();
    for(copyIndex=0;copyIndex<SCOUNT;copyIndex++)
    analogBufferTemp[copyIndex]= analogBuffer[copyIndex];
    averageVoltage = getMedianNum(analogBufferTemp,SCOUNT) * (float)VREF/ 1024.0; // read the analog value more stable by the median filtering algorithm, and convert to voltage value
    float compensationCoefficient=1.0+0.02*(temperature-25.0); //temperature compensation formula: fFinalResult(25^C) = fFinalResult(current)/(1.0+0.02*(fTP-25.0));
    float compensationVolatge=averageVoltage/compensationCoefficient; //temperature compensation
    // LE 00.595 EST MON CALCUL ICI !!
    
    tdsValue=1.547*(133.42*compensationVolatge*compensationVolatge*compensationVolatge - 255.86*compensationVolatge*compensationVolatge + 857.39*compensationVolatge)*0.5; //convert voltage value to tds value
    return tdsValue;
    
//Serial.print("voltage:");
//Serial.print(averageVoltage,2);
//Serial.print("V ");
//Serial.print("TDS Value:");
//Serial.print(tdsValue,0);
//Serial.println("ppm");
}
}


void serial_flush(void) {
 while (true)
   {
   delay (20);  // give data a chance to arrive
   if (Serial.available ())
     {
     // we received something, get all of it and discard it
     while (Serial.available ())
       Serial.read ();
     continue;  // stay in the main loop
     }
  else
    break;  // nothing arrived for 20 ms
   }
}
6

ESP

ESP

/*

#include <WiFi.h>
#include <FirebaseESP32.h>
#include <ArduinoJson.h>



#define FIREBASE_HOST XXX
#define FIREBASE_AUTH XXX
#define WIFI_SSID XXX
#define WIFI_PASSWORD XXX

    // Create a Json Buffer that will receive the data from the Arduino
  DynamicJsonBuffer jb;
    // Create the request message that will be sent to the arduino
  JsonObject& req = jb.createObject();
  
//Variables
float lux;
float dist;
float humidity;
float waterlevel;
float temp;
float tdsValue;
float watertemp;
float dimlightvalue;

//Define FirebaseESP32 data object
FirebaseData firebaseData;
int i;
int pumpstate;

FirebaseJson json;

void printResult(FirebaseData &data);


void setup()
{
  i=0;
  
  Serial.begin(115200);

  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  Serial.print("Connecting to Wi-Fi");
  while (WiFi.status() != WL_CONNECTED)
  {
    i++;
    Serial.print(".");
    delay(300);
    if(i>10){
      ESP.restart();
    }
  }
  Serial.println();
  Serial.print("Connected with IP: ");
  Serial.println(WiFi.localIP());
  Serial.println();

  Firebase.begin(FIREBASE_HOST, FIREBASE_AUTH);
  Firebase.reconnectWiFi(true);
}



void loop()
{
   tryagain:// Used to send another request if no answer comes

  Firebase.getInt(firebaseData, "/Request/Requestall");
  int requestdata=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/UpdateLight");
  int requestupdatelight=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/PumpChange");
  int requestpump=firebaseData.intData();

  while(requestdata == 0 && requestupdatelight==0 && requestpump==0) {// Block while there is no request from user
    
  delay(1000);

  Firebase.getInt(firebaseData, "/Request/Requestall");
  requestdata=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/UpdateLight");
  requestupdatelight=firebaseData.intData();

  Firebase.getInt(firebaseData, "/Request/PumpChange");
  requestpump=firebaseData.intData();

    
    }

 if(requestdata>0){
   req["type"] = "request";

   Firebase.getInt(firebaseData, "/Control/DimLight"); // Pour corriger le bug
   dimlightvalue=firebaseData.intData(); 
   req["lightdim"] = dimlightvalue;

   req.printTo(Serial); 

 boolean messageReady = false;
 String message = "";
 int i=0;
 
  while(messageReady == false) { // Runs until a message is received back
    
      if(Serial.available()) {
       message = Serial.readString(); // Read the message
       messageReady = true;  // "Break" the while and continue
       continue; //skip the delay if necessary
     }
     
     delay(10);
     i++;
     if(i % 1000 == 0) {goto tryagain;}// If after 10s nothing comes back, tries again to send a request
  }

    JsonObject& obj = jb.parseObject(message); // Translate the received message into a JsonObject for treatment
    lux = obj["lux"];
    dist= obj["dist"];
    temp= obj["temp"];
    humidity= obj["humidity"];
    waterlevel= obj["waterlevel"];
    tdsValue=obj["tdsvalue"];
    watertemp=obj["watertemp"];
    obj.prettyPrintTo(Serial);
    Serial.println();

    Firebase.setDouble(firebaseData,"Sensors/Light", lux);
    Firebase.setDouble(firebaseData,"Sensors/TankHeight", dist);
    Firebase.setDouble(firebaseData,"Sensors/AirTemp", temp);
    Firebase.setDouble(firebaseData,"Sensors/Humidity", humidity);
    Firebase.setDouble(firebaseData,"Sensors/WaterLevel", waterlevel);
    Firebase.setDouble(firebaseData,"Sensors/EC", tdsValue);
    Firebase.setDouble(firebaseData,"Sensors/WaterTemp", watertemp);


    
    Firebase.setDouble(firebaseData,"Request/Requestall", 0); // Reset the request
    requestupdatelight=0;
 }


 if (requestupdatelight>0){

Firebase.getInt(firebaseData, "/Control/DimLight");
dimlightvalue=firebaseData.intData();

   req["type"] = "control";
   req["lightdim"] = dimlightvalue;
   req.printTo(Serial); // Send the message


   Firebase.setDouble(firebaseData,"Request/UpdateLight", 0); // Reset the request

}

 if (requestpump>0){ 
  
req["type"] = "control";

  Firebase.getInt(firebaseData, "/Control/PumpState");
  pumpstate=firebaseData.intData();

  req["pumpstate"] = pumpstate ; // Reads the pump state (dictated by the app)
    


   req.printTo(Serial); // Send the message


  Firebase.setDouble(firebaseData,"/Request/PumpChange", 0); // Reset the request

}


}
7

Do not use the "String" object on the Arduino, it causes memory corruption and that will defunct the Arduino.

8

What would you advice ?

9

Avoid using JSON as protocol between ESP32 and Arduino. Use raw / binary data in a "struct" or a string with start/end marker, see this guide.

10

Thank you, it is very interesting.
But I am keen on using the Json libraries, at it is simple. Your post seems to use a more "raw" method. Do you any other methods that would use the Json method ?