[Solved] Trouble in JSON Deserializing

Projects Programming
1

Hi all,

I've got a problem when trying to deserialize a JSON received from Serial port. I've tried many ways but now I'm quite out of solution.

Here is the received JSON (I've raised my Serial buffer size to 256):

{"sensors": 2,"labels": ["garage","exterieur"],"values": [48.75608,2.302038],"heure":12,"minute":35,"seconde":50,"jour":12,"mois":2,"annee":2020}

This JSON is sent by another Arduino, I've checked the output and it is ok. For debugging purpose, I send it now with CoolTerm.

Here is how I currently (I've changed this part many times in order to find the problem) manage the Serial input:

void serialEvent(){
 packet_size = 0;

 if (Serial.available()){
 memset(serial_packet_char,0,sizeof(serial_packet_char));
 initialiseTFT();
 
 
 tft.setTextSize(1);
 tft.setTextColor(color_text);

 tft.setCursor(5,20);
 tft.print("Buffer : ");
 tft.print(SERIAL_RX_BUFFER_SIZE);
 tft.println(" bytes");

 while(Serial.available()){
 packet_size++;
 serial_packet_char[packet_size-1] = (char)Serial.read();
 }
 serial_packet_char[packet_size] = (char)0;

 tft.setCursor(5,30);
 tft.print("Received : ");
 tft.print(packet_size);
 tft.println(" bytes");

 if (packet_size > 0){new_packet_received = true;}
 }
}

The serial_packet_char is defined in the head of the program like this:

char serial_packet_char[SERIAL_RX_BUFFER_SIZE];

And same for the JSON document:

const size_t capacity = 2*JSON_ARRAY_SIZE(NB_SENSORS) + JSON_OBJECT_SIZE(9);
DynamicJsonDocument packet(capacity);

The new_packet_received variable is handled in the loop, and call the following function if true:

void handleNewPacket(){
 Serial.println("Handling new packet, first cleaning");
 packet.clear();
 new_packet_received = false;

 //DEBUG
 Serial.print("-");
 Serial.print(serial_packet_char);
 Serial.println("-");

 Serial.println("Now deserialize");
 DeserializationError err = deserializeJson(packet, serial_packet_char);
 Serial.println("End of deserialize");
}

The program run well until I send the JSON from CoolTerm. When I do that, I see properly the packet size on the TFT screen, but when the call to deserializeJson is done, the Arduino is rebooting.

Serial output when commenting the call to deserializeJson:

Start setup
PING
PING
PING
New packet ready to be handled
Handling new packet, first cleaning
-{"sensors": 2,"labels": ["garage","exterieur"],"values": [48.75608,2.302038],"heure":12,"minute":35,"seconde":50,"jour":12,"mois":2,"annee":2020}-
PING
PING

Serial output when uncommenting the call to deseralizeJson:

Start setup
PING
PING
PING
New packet ready to be handled
Handling new packet, first cleaning
-{"sensors": 2,"labels": ["garage","exterieur"],"values": [48.75608,2.302038],"heure":12,"minute":35,"®*WÉсsetup
PING
PING

We can see that we have the print of the JSON string, but we don't have the "Now deserialize" message, we jump directly to the "start setup" string (which is in the setup block), and the message is truncated with ascii artefacts.
The "PING" are just 2 seconds ping done by my program to tell the other Arduino that this one still alive.

In addiction, I specify that after compilation of my program, the memory is quite full:

Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
DATA:    [========= ]  90.0% (used 1844 bytes from 2048 bytes)
PROGRAM: [========  ]  75.0% (used 24196 bytes from 32256 bytes)

Do you have any clue ? I've spent 2 days on that, and I am out of solutions...

2

Post the whole thing - attach it if it's too big.

It looks like you're making the common mistake of assuming that if there's serial data to process that all of it has arrived. That's rarely true, but your serial output does seem to be ok.

Lack of free memory is what I'd focus on. Add the F macro to your serial prints to get their text from progmem.

3

You were right about free memory.

I have used the F macro on each hardcoded labels and commented many features I don't need for now, in order to keep some spare SRAM.

Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
DATA:    [====      ]  39.1% (used 801 bytes from 2048 bytes)
PROGRAM: [========  ]  82.4% (used 26580 bytes from 32256 bytes)

Now the deserialization function doesn't crash the Arduino.

After that, I have also changed the way I deserialize by passing directly the Serial object into the method. I got the "NoMemory" error on my TFT screen. That was because I forgot the "Additional bytes for strings duplication" in my capacity constant. I've corrected it:

const size_t capacity = 2*JSON_ARRAY_SIZE(NB_SENSORS) + JSON_OBJECT_SIZE(9) + 76;		// Calcul taille mémoire du JSON

Now the deserialize method doesn't crash and don't return any error. But the problem is not solved anyway.
Here is my new SerialEvent function:

void serialEvent(){
	if (Serial.available()){
		initialiseTFT();
		
		Serial.println(F("Now deserialize"));
		DeserializationError err = deserializeJson(packet, Serial);
		Serial.println(F("End of deserialize"));

		if (err) {
			tft.setTextSize(2);
			tft.setCursor(0,80);
			tft.setTextColor(convertColor(0,255,255));
			tft.println(F("deserializeJson() failed with code: "));
			tft.println(err.c_str());
			delay(4000);
			emptySerialBuffer();
		}
		else{
			serializeJson(packet,Serial);
			Serial.println("");
			Serial.print(F("Heure : "));
			Serial.println((int)packet['heure'].as<int>());
		}
	}
}

As you can see, after deserializing the packet, I serialize it to the Serial output in order to check if the object content is correct... and it is !!!

Start setup
PING
PING
Now deserialize
End of deserialize
{"sensors":2,"labels":["garage","exterieur"],"values":[48.75608,2.302038],"heure":12,"minute":35,"seconde":50,"jour":12,"mois":2,"annee":2020}
Heure : .
PING
PING

However, I still can't access to its value. Instead of getting the hour integer (which is 12), I receive a dot "."
The result is the same if I let the implicit type:

			Serial.println("");
			Serial.print(F("Heure : "));
			Serial.println((int)packet['heure']);

I'm pretty sure I am doing something wrong with the Json document but I don't find what...

4

Use double quotes around heure rather than the single ones you're currently using.

5

Thank you, I haven't realized that... shame on me. It is working with a simple cast:

Serial.print((int)packet["heure"]);

But it's not working with the non-implicit method:

Serial.print((int)packet['heure'].as<int>());

However it doesn't work with characters, this:

Serial.println((char)packet["labels"][0]);

Returns me, that:

{"sensors":2,"labels":["garage","exterieur"],"values":[48.75608,2.302038],"heure":12,"minute":35,"seconde":50,"jour":12,"mois":2,"annee":2020}
Labels : .
6

Sorry, it was because I wasn't casting it as a pointer:

Serial.println((char*)packet["labels"][0]);

I have to wash my mind from that, I was stuck too long into this, and do beginner mistakes.

Now all is working fine. Thank you very much WildBill !