Errors on Serial (even with 9600 baud)

Using Arduino Programming Questions
1

Hi guys,

I need your help. I try to do serial communication between an arduini Mega and an ESP8266 (or in my case a NodeMCU)

All in all the code works but I some lines in the serial are missing always.

for example I try to send:

 <toESP|pH_7.02|*014>
 <toESP|tE_25.63|*015>
 <toESP|tV_0|*011>
 <toESP|cI_0|*011>
 <toESP|mO_0|*011>
 <toESP|mM_0|*011>
 <toESP|cP_224.00|*016>
 <toESP|cR_0.00|*014>
 <toESP|cG_0.00|*014>
 <toESP|cB_0.00|*014>
 <toESP|p1_0|*011>
 <toESP|p2_0|*011>
 <toESP|lV_0|*011>
 <toESP|l1_0|*011>
 <toESP|l2_0|*011>
 <toESP|cO_1|*011>
 <toESP|hV_0|*011>
 <toESP|d1_1|*011>
 <toESP|d2_1|*011>
 <toESP|d3_1|*011>
 <toESP|cV_0|*011>
 <toESP|nO_1458672841|*020>

but I receive

 toESP|pH_7.02|*014
 toESP|tV_0|*011*015
 toESP|tV_0|*011*015
 cI_0|*011
 toESP|mO_0|*011
 toESP|mM_0|*011
 toESP|cP_224.00|*016
 toESP|cR_0.00|*014
 toESP|cB_0.00|*014
 oESP|p1_0|*011
 toESP|p2_0|*011
 toESP|lV_0|*011
 toESP|l1_0|*011
 toESP|l2_0|*011
 toESP|cO_1|*011
 toESP|hV_0|*011
 toESP|d1_1|*011
 toESP|d2_1|*011
 toESP|d3_1|*011
 toESP|cV_0|*011
 toESP|nO_1458672953|*020

The wierd thing is, that most of the time the second line is missing.

my code to send

void sendSerial(String sendCom)
{String sendThis = "<"+ sendCom + "|" ; 
 int CheckSum = sendThis.length()-1;
 sendThis+='*';
 if(CheckSum<10) {sendThis+="00";}
 else if (CheckSum<100) {sendThis+=0;}                
 sendThis+=CheckSum;    
 sendThis+=">";            
 char charBuf[200];
  sendThis.toCharArray(charBuf, 200);
  Serial2.write(charBuf);


  
# if debug
  {Serial.print(F("sendSerial DebugLine: "));
    Serial.print(charBuf);
  }
#endif

}

my code to receive

  void recvWithStartEndMarkers() {

 
  static boolean recvInProgress = false;
    char startMarker = '<';
    char endMarker = '>';
    char rc;
   
 
    while (Serial.available()) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx=0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }

}









  
void useNewData() {

    if (newData == true) {

                StringFromSerial=receivedChars;
                webSocket.broadcastTXT("dbg:"+String(StringFromSerial));
                parseCommand(StringFromSerial);
              // sendSerial("dbgES: " + StringFromSerial);
                StringFromSerial="";
                ndx = 0;
                newData = false;
              }
}

as you already know - in the loop the useNewData() and recvWithStartEndMarkers() is constantly called

Any ideas?

2

NatroN:
Any ideas?

Yes.
Post complete programs, not snippets.

Have you tested to see if your receive program is receiving the data without trying to do anything else with the data?

...R

3

hmm..

the code on the arduino is 6414 lines long
https://github.com/NatroNx/Aquaduino2_full

the code on the ESP8266 is 690 lines long

Will take some time to rewrite the code and remove everything not needed to reproduce my problem.

4

NatroN:
Will take some time to rewrite the code and remove everything not needed to reproduce my problem.

I don't mind waiting. It is much easier to help when the code is short.

If code won't fit then please it as attachments - not links to another website.

...R

5

That is a LOT of code for the Uno. To use all that code, AND piss away resources on the String class just does not make sense. Use a char array and sprintf() to prepare the string to send.

Send the string using Serial.print().

Check how much memory you have before populating the String and after extracting the string from the String. I suspect that the second answer is going to be "not enough".

6

Hi Paul,

thanks for the answer. I'm running that code on an arduino mega at about 70% used memory.

As Robin2 suggested I commented out everything not needed for testing. Nothin in the loop expect that Serial.send stuff.

What I also tried is to send the serial data to both - my serial. (which is my pc console) as well as the serial2. (which is the ESP8266)

As expected the serial output (which is running at a much higher baudrate than the ESP) is retuning everything perfect.

something like
Serial2.write(charBuf);
Serial.println(charBuf);

which makes me think that the problem may not be the arduino mega itself .. or not even the very short line to the ESP) - I think the problem may be the ESP which does something else then listening in that second. Becuase there are no missreads.. just missing characters.

i need to continue my search.

7

Just wanted to let you know I found and fixed my problems.

When you read my core on the ESP (second link esp.ino) you'll notice that recvWithStartEndMarkers() is called in the loop.

This function reads the data whenever something is available via Serial with Serial.read() blocking other function while receiving.

So what happened in my case?

Whenever I send many lines of data recvWithStartEndMarkers() should read one line and after that useNewData() should parse that line. But for me, when the String Received was very long it happened that the String was received but NOT parsed with useNewData(). In the next loop the receive function got the next line without using the first one.

So the fix was quite easy. Whenever a string is received the boolean newData variable is set to true. Once this is true, there is NO serial.read() until the useNewData() method parsed that data and set the boolean newData to false.

So what can happen now - as no serial.read() happens while the old String is not processed I may miss some information via Serial. But the useNewData() just parses a String and writes to a variable. That should be done very very fast. At least I had no issue while testing - and I tested a lot.

Thanks for your tipps!

8

NatroN:
This function reads the data whenever something is available via Serial with Serial.read() blocking other function while receiving.

The function recvWithStartEndMarkers() will return very quickly - the longest it will take is the time needed to read 64 characters that are already in the Serial Input Buffer. I doubt that it ever takes as long as 1 millisec.

Without seeing your latest code I can't understand your other comments.

9

already updated my git

10

NatroN:
already updated my git

I don't understand ?

...R

11

Github - you can see the whole code - every line I changed...

or have a look at this. I tried to post the code but only 9000 chars are allowed so i kicked out unimportant functions
while (Serial.available() > 0 && newData == false)

here I added && newData==false

//v.1.0.0 - 08.03.2016 -   init 
//v.1.0.1 - 08.03.2016 - minor fixes
//v.1.0.2 - 10.03.2016 - sendSerial with and without Feedback  - choose your weapons
//v 1.0.3 - 25.03.2016 - fixed the bug that Information received over serial wasn't complete
//v 1.0.4 - 28.03.2016 - changed ntp to europe (for DST), added automatic DST switching

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include <ESP8266HTTPUpdateServer.h>
#include <Arduino.h>
#include <WebSocketsServer.h>
#include <Hash.h>
#include "RTClib.h"
#include <Wire.h>
#include <WiFiUdp.h>
#include <pgmspace.h>


#define BLUEPIN 13

#define SSID_ME "Wlan07"
#define PW_ME "w_.."
#define HOST_ME "ESP8266"



unsigned int localPort = 2390;      // local port to listen for UDP packets
IPAddress timeServerIP; // time.nist.gov NTP server address
const char* ntpServerName = "europe.pool.ntp.org";
const int NTP_PACKET_SIZE = 48; // NTP time stamp is in the first 48 bytes of the message
byte packetBuffer[ NTP_PACKET_SIZE]; //buffer to hold incoming and outgoing packets
// A UDP instance to let us send and receive packets over UDP
WiFiUDP udp;

#define debug 0




const byte numChars = 230;
char receivedChars[numChars];
static byte ndx = 0;
boolean newData = false;

boolean webSockUpdate = false;
unsigned long webSockWait = 0;






enum {tPhWert, tTemp, tcalculatedPWM, tcalculatedRed, tcalculatedGreen, tcalculatedBlue, tTVModeState, tcleaningInProcess,
      tmanualOverride, tMoonModeState, tpump1Value, tpump2Value, tlight230Value, tlight1Value,
      tlight2Value, tco2Value, theaterValue, tdPump1Value, tdPump2Value, tdPump3Value, tcoolValue, tnow, tprocess
     };

const char PhWert_Char[] PROGMEM    = "pH";
const char Temp_Char[] PROGMEM    = "tE";
const char calculatedPWM_Char[] PROGMEM    = "cP";
const char calculatedRed_Char[] PROGMEM    = "cR";
const char calculatedGreen_Char[] PROGMEM    = "cG";
const char calculatedBlue_Char[] PROGMEM    = "cB";
const char TVModeState_Char[] PROGMEM    = "tV";
const char cleaningInProcess_Char[] PROGMEM    = "cI";
const char manualOverride_Char[] PROGMEM    = "mO";
const char MoonModeState_Char[] PROGMEM    = "mM";
const char pump1Value_Char[] PROGMEM    = "p1";
const char pump2Value_Char[] PROGMEM    = "p2";
const char light230Value_Char[] PROGMEM    = "lV";
const char light1Value_Char[] PROGMEM    = "l1";
const char light2Value_Char[] PROGMEM    = "l2";
const char co2Value_Char[] PROGMEM    = "cO";
const char heaterValue_Char[] PROGMEM    = "hV";
const char dPump1Value_Char[] PROGMEM    = "d1";
const char dPump2Value_Char[] PROGMEM    = "d2";
const char dPump3Value_Char[] PROGMEM    = "d3";
const char coolValue_Char[] PROGMEM    = "cV";
const char now_Char[] PROGMEM    = "nO";
const char process_Char[] PROGMEM    = "pR";

PGM_P const Char_table[] PROGMEM  =
{ PhWert_Char, Temp_Char, calculatedPWM_Char, calculatedRed_Char, calculatedGreen_Char, calculatedBlue_Char, TVModeState_Char,
  cleaningInProcess_Char, manualOverride_Char, MoonModeState_Char, pump1Value_Char, pump2Value_Char,
  light230Value_Char, light1Value_Char, light2Value_Char, co2Value_Char, heaterValue_Char,
  dPump1Value_Char, dPump2Value_Char, dPump3Value_Char, coolValue_Char, now_Char, process_Char
};

int charCount = sizeof(Char_table) / sizeof(Char_table[0]);







IPAddress ip(10,0,0,200);  //Node static IP
IPAddress gateway(10,0,0,138);
IPAddress subnet(255,255,255,0);


WebSocketsServer webSocket = WebSocketsServer(9000);
ESP8266WebServer httpServer(80);
ESP8266HTTPUpdateServer httpUpdater;
const char* host = HOST_ME;
const char* ssid     = SSID_ME;
const char* password = PW_ME;
unsigned long lastTimeHost;
unsigned long lastTimeSent;
unsigned long lastTimeNTPUpdate;
String text="z10";






// time to get live
DateTime now;
//modes
boolean TVModeState = false;
boolean cleaningInProcess = false;
boolean manualOverride = false;
boolean MoonModeState = false;
//lights
float calculatedPWM = 100;   
float calculatedRed = 1;
float calculatedGreen = 2;
float calculatedBlue = 3;
//Relais
boolean pump1Value = false;
boolean pump2Value = false;
boolean light230Value = true;
boolean light1Value = true;
boolean light2Value = true;
boolean co2Value = true;
boolean heaterValue = false;
boolean dPump1Value = true;
boolean dPump2Value = true;
boolean dPump3Value = true;
boolean coolValue = true;
//values to monitor
float PhWert = 7.01; //sting to float to calculate with it
float Temp = 11.11;  






String StringFromSerial;
unsigned long sent = millis();
int i = 0;
int callme=0;




void setup()
{
  Serial.begin(9600);     
  Serial.println("Serial Monitor Connected");  
  pinMode(BLUEPIN, OUTPUT);
  analogWriteFreq(200);
  WifiConnect();
  WebSocketConnect();
  HTTPUpdateConnect();
  udp.begin(localPort);     //ntp 
  lastTimeNTPUpdate=0;
}





  void recvWithStartEndMarkers() 
{  static boolean recvInProgress = false;
    char startMarker = '<';
    char endMarker = '>';
    char rc;
   
    while (Serial.available() > 0 && newData == false)     //----------- this is the line i changed ------
    {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx=0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }

}









  
void useNewData() {

    if (newData == true) {

                StringFromSerial=receivedChars;
                
                parseCommand(StringFromSerial);
              // sendSerial("dbgES: " + StringFromSerial);
                StringFromSerial="";
                ndx = 0;
                newData = false;
              }
}








void loop() 
{  
   recvWithStartEndMarkers();
    useNewData();
}
12

I didn't look at all the code, but the first examples showing sent vs received, looks like some lines are missing, and others are duplicated.
At first glance in your initial code snippets - do you manage 'clearing' the rx buffer after each complete message? or is it just happening by coincidence?

Also as suggested earlier - lose the Arduino string functions and get to know the C functions to manipulate your char[] arrays. Faster, more flexible, and very powerful. Well worth the learning curve.

13

NatroN:
or have a look at this. I tried to post the code but only 9000 chars are allowed so i kicked out unimportant functions
while (Serial.available() > 0 && newData == false)

here I added && newData==false

I avoid Github as much as I can (one of the perks of advancing years :slight_smile: )

The line while (Serial.available() > 0 && newData == false) is in my example code so I don't know what question you are trying to ask - if any?

...R