Best way to make sure all serial data is read

[jbsmith03]

I am sending serialized json data from my arduino to a program that is running in C++ but am getting some incomplete data. I was using a temporary fix checking whether the object!= NULL which was working but meant that I was losing some data.

The C++ code is using the serialPort library and my DATA_LENGTH is 64.

char receivedData[DATA_LENGTH];
   	if (arduino->isConnected()) {
   		int hasRead = arduino->readSerialPort(receivedData, DATA_LENGTH);
   		if (hasRead) {
   			DEBUG(receivedData);
   			json_t* object = json_loads(receivedData, DATA_LENGTH, NULL);
   			//if (object != NULL) {
   				json_t* value = json_object_get(object, "v");

The arduino program is sending the data of a 10k pot. Here is my loop()

void loop(void) {
  enc();
  total = total - readings[readIndex];
  readings[readIndex] = map(analogRead(inputPin), 0, 1023, 0, 1000);
  total = total + readings[readIndex];
  readIndex = readIndex + 1;

  if (readIndex >= numReadings) {
    readIndex = 0;
  }

  average = total / numReadings;

  if (abs(average-old_average)>=3){
    
    DynamicJsonDocument doc(64);
    doc["m"] = 3;
    doc["p"] = rotaryPosition;
    doc["v"] = average;
    serializeJson(doc, Serial);
   
    //static char str[8];
    //Serial.write(str, sizeof(str));
    
    tft.setArcParams(1180);
    tft.setAngleOffset(210);
    
    tft.fillArc(60, 60, 40, 8, 0, average, ILI9341_BLUE);
    tft.fillArc(60, 60, 40, 8, average, 1000, ILI9341_WHITE);
    
    old_average = average;
  }
 
  
  delay(1);
}

At the moment I have a DEBUG() in the C++ code printing the receivedData to a file. Here is what I am getting

Wondering what is the best way to make sure I have the complete serialized Json object?

[Koepel]

I don't see anything missing. Sometimes your receive a single JSON package in two chunks. You have to know where the begin and where the end of the data package is.

One way to do that is to read every single received byte into a buffer. Check the buffer to see if the data is complete.

[jbsmith03]

Thank you. So because I know that I'm only sending a single object I can append the chunks to a separate buffer then check if the last character of the buffer is "}" ?

[Koepel]

Yes, and you can start reading when a '{' is received.

When you try to read the full DATA_LENGTH at once, that data might not be complete and might still be coming in. There could also be a synchronization problem (reading the last part of a package and the first part of the next package) because once you are out of sync, there is no code to get in sync again.

[jbsmith03]

Updated the code to read into another buffer but something is still wrong

		char receivedData[DATA_LENGTH];
		if (arduino->isConnected()) {
			int hasRead = arduino->readSerialPort(receivedData, DATA_LENGTH);
			if (hasRead) {
				int receivedData_length = strlen(receivedData);
				for (int i=0; i <= receivedData_length; i++) {
					int fullObject_length = 0;
					fullObject_length = strlen(fullObject);
					fullObject[fullObject_length + i] = receivedData[i];
				}
				char end_char = fullObject[strlen(fullObject) - 1];

				if (end_char == '}') {
					DEBUG(fullObject);

[jbsmith03]

forgot to mention I initialise my fullObject out of the loop and clear it after my debug statement.

[Koepel]

I don't know what those functions do and how large the buffers are. You could print the 'receivedData' after the call to readSerialPort() ?

Is the readSerialPort() your own function ? How does it know when to stop reading ? Does it have a timeout ? Does it always add a zero-terminator at the end ?

[jbsmith03]

The serialPort function is from a library that i have imported.

GitHub - manashmandal/SerialPort: C++ Library for Serial Communication with...

Think I might need to dig into the library file to find out exactly whats going on

[Koepel]

At first glance I think that the library is not fail safe.

A memset() fills the buffer with zeros, hopefully one of those zeros will become a zero-terminator for the data. If there is enough incoming data to fill the whole buffer, then the buffer will be filled up to the very last byte and then there is no zero-terminator at the end.
Suppose you read a lot of rubbish because of some bug and all 64 bytes will be filled, then your code will fail.

This is the explanation of the used COMSTAT: https://docs.microsoft.com/en-us/windows/win32/api/winbase/ns-winbase-comstat.

Let's assume that your 64 byte buffer is large enough for the small JSON data.
Then it is possible that not everything is read. You could print the 'hasRead' value.

It is long time ago that I used Windows/Microsoft programming. Can you debug it or print variables right after the readSerialPort() call ?

There is probably a bug that causes the first character and the last-1 to be wrong, but I don't see that bug.

[jbsmith03]

Feel like I might be doing something wrong when I'm trying to append to the fullObject buffer as It looks like it is being read in the correct order as it looks fine when I print the 2 chunks from receivedData in the first screen shot.

[jbsmith03]

Got there in the end!

char receivedData[DATA_LENGTH];
		if (arduino->isConnected()) {
			int hasRead = arduino->readSerialPort(receivedData, DATA_LENGTH);
			if (hasRead) {
				std::strcat(fullObject, receivedData);
				if (fullObject[strlen(fullObject)-1] =='}') {
					DEBUG(fullObject);
					fullObject[0] = 0;

Thanks for your help