Incomplete String Over Serial Transfer

JasperJP · April 1, 2019, 2:22pm

I'm trying to send a string over a Serial connection to my NodeMcu esp8266 module (coming from an arduino).

This is the code that I currently have on my esp8266:

#include <ESP8266WiFi.h>;
#include <ESP8266HTTPClient.h>;
#include <SoftwareSerial.h>


SoftwareSerial NodeMCU(D7,D8);
 
const char* ssid = "Hello";
const char* password = "hello123";
 
void setup () {
   pinMode(D7,INPUT);
   pinMode(D8,OUTPUT);
   
   Serial.begin(9600);
   NodeMCU.begin(9600);
   
   WiFi.begin(ssid, password);

}
 
void loop() {
  
    String str = "";
    if (NodeMCU.available()) {
    
    char c;
    while ((c = NodeMCU.read()) != 255) {
    if (c != 255) {
        str.concat(c);
      }
    }
    Serial.println(str);
    }


   
   
  if (WiFi.status() == WL_CONNECTED) { 
   
     HTTPClient http; 
   
     http.begin("http://mysite.com/api/1"); 
     http.addHeader("Content-Type", "application/json"); 
     int httpCode = http.PUT(str); 
     String payload = http.getString(); 
     http.end();
    }
   
 
}

If I execute this code when my module is NOT connected to wi-fi, I receive in the serial:

{"id":1,"temp":22.9,"hum":48,"lightlevel":"12","lightstatus":"ON","humgoal":"10"}

BUT when I execute this code when my module IS connected, i receive an incomplete String in the serial:

{"id":1,"temp":22.9,"hum":48,"lightlevel":"12","lightstatus":"O

Why does this happen and how can I make sure that I receive all data?
Thanks

wildbill · April 1, 2019, 2:36pm

The serial port is buffered and can keep 64 characters waiting for you to read them. After that they're thrown away. You're sending 81, so if you're not reading it fast enough, some will be dropped, as you observe.

The http activity you're doing takes time so if that code is running while your serial data is being sent, you're not reading the serial port, hence the overflow.

Also, your code assumes that when it starts reading the string, it will get all the characters in one pass through loop. You would be better off using the braces you've got for start & end markers to let you revisit loop multiple times until you have the whole packet.

system · April 1, 2019, 2:40pm

Why does this happen and how can I make sure that I receive all data?

You are getting less than a full packet because your only criteria for stopping reading, or continuing to read, is that the incoming serial buffer is empty.

That is a piss-poor definition of a complete packet. Whatever is sending the data should do a better job, using start and end of packet markers, so the reader KNOWS when a packet starts and when it ends, and doesn't try to use a partial packet, so some mess that contains more than one packet, or parts of two different packets.

Serial Input Basics - Updated is a good place to start.

JasperJP · April 3, 2019, 12:25pm

I've tried example 2 and 3 from Serial Input Basics - updated - Introductory Tutorials - Arduino Forum but still receive the incomplete data.

system · April 3, 2019, 12:26pm

JasperJP:
I've tried example 2 and 3 from Serial Input Basics - updated - Introductory Tutorials - Arduino Forum but still receive the incomplete data.

Show us the code you are actually using, on both ends of the serial connection.

JasperJP · April 3, 2019, 12:33pm

This is the String that i'm trying to send: (Arduino code)

ArduinoUno.println("<{\"id\":1,\"temp\":20.5,\"hum\":80,\"lightlevel\":667,\"lightstatus\":\"ON\",\"humgoal\":70}>");

This is the receiving end: (NodeMcu esp8266)

#include <ESP8266WiFi.h>;
#include <ESP8266HTTPClient.h>;
#include <SoftwareSerial.h>

const byte numChars = 32;
char receivedChars[numChars];

boolean newData = false;

SoftwareSerial NodeMCU(D7,D8);
 
const char* ssid = "Hello";
const char* password = "hallo123";
 
void setup () {
   pinMode(D7,INPUT);
   pinMode(D8,OUTPUT);
   
   Serial.begin(9600);
   NodeMCU.begin(9600);
   
   WiFi.begin(ssid, password);
}


void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;
 
    while (NodeMCU.available() > 0 && newData == false) {
        rc = NodeMCU.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

void showNewData() {
    if (newData == true) {
        Serial.print("This just in ... ");
        Serial.println(receivedChars);
        newData = false;
    }
}

 
void loop() {
  
    recvWithStartEndMarkers();
    showNewData();

   
   
  if (WiFi.status() == WL_CONNECTED) { 
   
     HTTPClient http; 
   
     http.begin("http://mysite.com/api/1"); 
     http.addHeader("Content-Type", "application/json"); 
     int httpCode = http.PUT("test"); 
     String payload = http.getString(); 
     http.end();
    }
   
 
}

KASSIMSAMJI · April 3, 2019, 12:34pm

be alerted too , what you see on serial monitor ,sometimes, is not what's in the variable string, it's cut short somewhat

the only way to ensure this is check on server side whether the API receives complete string or not

otherwise you might want to add a delay(1); in your while loop

system · April 3, 2019, 12:36pm

How many characters are in the String you are sending? How many have you defined as the maximum size of a packet?

JasperJP · April 3, 2019, 12:40pm

94 characters i believe. Where can I see/change the maximum size of a packet ?

system · April 3, 2019, 12:42pm

Where can I see/change the maximum size of a packet ?

const byte numChars = 32;
char receivedChars[numChars];

Change 32 to something large enough to hold your packet.

JasperJP · April 3, 2019, 12:47pm

Great that actually works BUT it still brings me back to the original problem. It ONLY works when the Wi-Fi is disconnected (when the last "if" is not executed).

wildbill · April 3, 2019, 1:01pm

I still suspect that your wifi code takes long enough to execute that the serial port buffer is becoming full and throwing stuff away. You could try printing millis before and after it to see if that's true.

An easy (though unsatisfying) fix would be to reduce the amount of text you're sending. The Arduino doesn't need those long human readable identifiers like "lightlevel", make them substantially smaller.

gfvalvo · April 3, 2019, 1:18pm

Why not implement some handshaking between the TX and RX? Don’t do the http stuff in the RX until you receive the complete packet. Don’t let the TX send the next packet until it receives an “all clear” message from the RX.

Idahowalker · April 3, 2019, 1:31pm

When I am using serial I use a protocol for sending and receiving. I start off by using a sentence. The sentence has a beginning, an instruction word, word separators, and a sentence end. I use a "<" as a sentence start, ">" sentence end, "," as a word separator. I am able to receive 255 characters with accuracy.

Here is the code I use to receive on an ESP32:

void fReceiveSerial_LIDAR( void * parameters  )
{
  bool BeginSentence = false;
  sSerial.reserve ( StringBufferSize300 );
  char OneChar;
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtReceiveSerial_LIDAR, pdTRUE, pdTRUE, portMAX_DELAY);
    if ( LIDARSerial.available() >= 1 )
    {
      while ( LIDARSerial.available() )
      {
        OneChar = LIDARSerial.read();
        if ( BeginSentence )
        {
          if ( OneChar == '>')
          {
            if ( xSemaphoreTake( sema_ParseLIDAR_ReceivedSerial, xSemaphoreTicksToWait10 ) == pdTRUE )
            {
              xQueueOverwrite( xQ_LIDAR_Display_INFO, ( void * ) &sSerial );
              xEventGroupSetBits( eg, evtParseLIDAR_ReceivedSerial );
              //
            }
            BeginSentence = false;
            break;
          }
          sSerial.concat ( OneChar );
        }
        else
        {
          if ( OneChar == '<' )
          {
            sSerial = ""; // clear string buffer
            BeginSentence = true; // found begining of sentence
          }
        }
      } //  while ( LIDARSerial.available() )
    } //if ( LIDARSerial.available() >= 1 )
    xSemaphoreGive( sema_ReceiveSerial_LIDAR );
  }
  vTaskDelete( NULL );
} //void fParseSerial( void * parameters  )

Notice the if ( LIDARSerial.available() >= 1 ) is a departure from the if ( LIDARSerial.available() >= 0 ). I found with the Due, STM32, and the ESP32 using a 1 is more reliable then using a 0. The event trigger for the code works well in the 1mS to .25mS range.

JasperJP · April 3, 2019, 1:35pm

Thank you very much for all the replies.
It was very usefull and i'm sure i'll make it work

Topic		Replies	Views
Arduino UNO to NodeMCU serial communication Programming Questions	12	787	September 17, 2023
No Output On The Serial Monitor Project Guidance	4	1277	May 5, 2021
readString() missing characters Programming Questions	5	1902	May 5, 2021
Blank screen On The Serial Monitor Programming Questions	61	5564	May 5, 2021
ESP8266 to arduino Programming Questions	6	583	May 5, 2021

Incomplete String Over Serial Transfer

Related Topics