Incorrect data transferring via UART

Hi.

I have two Mega2560 boards. One of them has Senseair S8 sensor for measuring CO2 level. Also one of them has Ethernet shield. Everything had been working great for several months till I wanted to connect the second board to the first one via Serial port.
So now I have this configuration: one board has connected Senseair via hardserial port. Another board requests data from the first one. I reduced my scatches to minimal reproducing form.
See below please:
Scetch of Mega with Senseair:

byte senseAirReadCO2Bytes[] = {0xFE, 0X44, 0X00, 0X08, 0X02, 0X9F, 0X25};
byte senseAirResponseBytes[] = {0, 0, 0, 0, 0, 0, 0};

const int senseAir_WAIT_REQUEST_STATE = 0;
const int senseAir_SEND_REQUEST_STATE = 1;
const int senseAir_RECEIVE_RESPONSE_STATE = 2;

int senseAirCurrentState = senseAir_SEND_REQUEST_STATE;
int senseAirReceiveTimeoutCount = 0;

String serial1SendingDataInputCommanString = "";
bool serial1SendingDataInputCommanStringComplete = false;


void setup() {
  Serial.begin(9600);
  while(!Serial); // time to get serial running

  Serial.println(F("Initializing Serial1 (18, 19 pins)..."));
  Serial1.begin(9600);
  while(!Serial1);

  Serial3.begin(9600);
  while(!Serial3);
}

void loop() {
  sendRequest(senseAirReadCO2Bytes);
  unsigned long valCO2 = getValue(senseAirResponseBytes);
  //valCO2 = 746;
  Serial.print("Co2 ppm = ");
  Serial.println(valCO2);

  Serial1SendingDataWorker(valCO2);
  
  delay(500);
}

void sendRequest(byte packet[]) { 
  if (!Serial3.available()) {
    for (int i = 0; i < 7; i++)
    {
      Serial3.write(senseAirReadCO2Bytes[i]);   // Push each char 1 by 1 on each loop pass
    }
    delay(50);
  }

  int timeout = 0;
  while (Serial3.available() < 7 ) {
    timeout++;
    if (timeout > 10) {
      while (Serial3.available())
        Serial3.read();
      break;
    }
    delay(50);
  }

  for (int i = 0; i < 7; i++)  {
    senseAirResponseBytes[i] = Serial3.read();
  }
}

unsigned long getValue(byte packet[]) {
  int high = packet[3];
  int low = packet[4];
  unsigned long val = high * 256 + low;
  return val;
}

void Serial1SendingDataWorker(unsigned long val) { 
  while (Serial1.available()) {
    char inChar = (char)Serial1.read();

    if (inChar != '\n')
      serial1SendingDataInputCommanString += inChar;
    
    if (inChar == '\n' && serial1SendingDataInputCommanString.length() > 0) {
      serial1SendingDataInputCommanStringComplete = true;
      Serial.print(F("Serial1SendingDataWorker: received command = "));
      Serial.println(serial1SendingDataInputCommanString);

      Serial1SendingData_SendResponse(val);
      return;
    }
  }

  if (serial1SendingDataInputCommanStringComplete)
    Serial1SendingData_SendResponse(val);
}

void Serial1SendingData_SendResponse(unsigned long val) {
  Serial.println(F("Serial1SendingData_SendResponse"));

  if (serial1SendingDataInputCommanString.equals("get")) {
    String jsonStr = String(val);

    for (int i = 0; i < jsonStr.length(); i++)
    {
      Serial1.print(jsonStr[i]);   // Push each char 1 by 1 on each loop pass
      Serial.print(jsonStr[i]);
    }
  } 

  Serial.println();
  serial1SendingDataInputCommanString = "";
  serial1SendingDataInputCommanStringComplete = false;
}

Scetch of the second Mega which requests data:

void setup() {
  Serial.begin(9600);
  while(!Serial); // time to get serial running

  Serial.println(F("Initializing Serial2 (16, 17 pins)..."));
  Serial1.begin(9600);
  while(!Serial1);
}

void loop() {
  SerialGettingDataWorker();
  
  delay(2000);
}

void SerialGettingDataWorker() {
  while (!Serial1.available()){
    Serial.println("Sending Serial2GettingData request");
    Serial1.write("get\n");
    delay(500);
  }

  if (Serial1.available()) {
    while(Serial1.available()){
      char q = Serial1.read();
      Serial.println(q);
      delay(50);
    }
    Serial.println();
  }
}

Initally I had big scatches which have a lot of data and sent then in JSON format. Now it's just example demostrating issue.

The problem is that requesting Mega gets numbers like 0 or 10000 or sometimes other figures and (what's very interesting for me) even symbols like "g" or "ge" or some corrupted symbol. I.e. it receives symbol from "get" that was sent to Mega with sensor!

At the same time if I uncomment

//valCO2 = 746;

line everything becomes perfect and the first controller gets 746 as it should be. Also I connected BME280 instead of Senseair and everything also worked perfect because BME280 does not user Serial.
So now I'm totally sure that problem is in using two Serials (actually even 3: Serial, Serial1 and Serial3) but I've spent already a lot of time and I don't know the nature of issue and how to resolve it.

P.S. Just in case. In Serial I see that I get correct values from Senseair (and when I used only requesting data via Ethernet everrything worked perfect - I was this correct CO2 value in JSON in my webbrowser). The more

Serial.print(jsonStr[i]);

shows that correct chars are sent. But on receiving side data is wrong.

Replace

  if (serial1SendingDataInputCommanString.equals("get")) {
    String jsonStr = String(val);

    for (int i = 0; i < jsonStr.length(); i++)
    {
      Serial1.print(jsonStr[i]);   // Push each char 1 by 1 on each loop pass
      Serial.print(jsonStr[i]);
    }
  }

by

  if (serial1SendingDataInputCommanString.equals("get")) {
    String jsonStr = String(val);
    Serial1.print(jsonStr);
    Serial.print(jsonStr);
  }

and report the results.

Your sketch is overly complex. If you restructure it, get rid of the use of the String class. It shouldn't be used on AVR-based Arduinos because it fragments the RAM too fast.

So now I'm totally sure that problem is in using two Serials (actually even 3: Serial, Serial1 and Serial3) but I've spent already a lot of time and I don't know the nature of issue and how to resolve it.

I'm quite sure that this isn't the source of your problems.

Hi,

Thanks for your help.

I tried as you suggested and replaced this part of code (though it was done initially in this way but I did byte-by-byte sending for debug purpose) - it does not work. I tried a lot of different variants because have been struggling with problem alredy more then one week:).

As mentioned earlier, your code is over-complex and "S"tring variables are always frowned upon in Arduinoland.

Something that will clear the code up a bit is to use a transfer library like SerialTransfer.h. This library automatically packetizes and parses serial data between arduinos. Its easy to use, fast, and reliable. It can be used to transfer individual bytes, ints, floats, arrays, and structs.

You can install the library through the Arduino IDE's Libraries Manager (search "SerialTransfer"). Example sketches can be found in the library after installation is complete.

If you want to "roll your own" serial transfer code instead or learn the simple fundamentals of serial transfer concepts, check out this tutorial.

I've already tried to use SerialTransfer today. It works with hardcoded value as it worked without this library. But it still does not work for case when being sent value is calculated based on data got from sensor.

hdimon:
It works with hardcoded value as it worked without this library. But it still does not work for case when being sent value is calculated based on data got from sensor.

You can transfer both hardcoded values and variables alike with the library. The only limitation on valid datatypes is that you can't transfer "S"trings, which, you shouldn't be using with Arduinos anyway.

Can you post your code when trying to use the library? I'm curious on how you tried to implement it, cause it certainly will work for your project. Do note that the library is ONLY used to interconnect Arduinos and is not intended to interface with sensors. You'll have to keep the Arduino <--> sensor code the same.

Long story short: Replace anything to do with "Serial3" with SerialTransfer.h code and keep the rest as you have it.

Sending board scetch:

#include "SerialTransfer.h"
SerialTransfer myTransfer;

byte senseAirReadCO2Bytes[] = {0xFE, 0X44, 0X00, 0X08, 0X02, 0X9F, 0X25};
byte senseAirResponseBytes[] = {0, 0, 0, 0, 0, 0, 0};

const int senseAir_WAIT_REQUEST_STATE = 0;
const int senseAir_SEND_REQUEST_STATE = 1;
const int senseAir_RECEIVE_RESPONSE_STATE = 2;

int senseAirCurrentState = senseAir_SEND_REQUEST_STATE;
int senseAirReceiveTimeoutCount = 0;

String serial1SendingDataInputCommanString = "";
bool serial1SendingDataInputCommanStringComplete = false;

void setup() {
  Serial.begin(9600);
  while(!Serial); // time to get serial running

  Serial.println(F("Initializing Serial1 (18, 19 pins)..."));
  Serial1.begin(9600);
  while(!Serial1);

  myTransfer.begin(Serial1);

  Serial3.begin(9600);
  while(!Serial3);
}

void loop() {
  sendRequest(senseAirReadCO2Bytes);
  unsigned long valCO2 = getValue(senseAirResponseBytes);
  //valCO2 = 746;
  Serial.print("Co2 ppm = ");
  Serial.println(valCO2);

  Serial1SendingDataWorker(valCO2);
  
  delay(2000);
}

void sendRequest(byte packet[]) { 
  if (!Serial3.available()) {
    for (int i = 0; i < 7; i++)
    {
      Serial3.write(senseAirReadCO2Bytes[i]);   // Push each char 1 by 1 on each loop pass
    }
    delay(50);
  }

  int timeout = 0;
  while (Serial3.available() < 7 ) {
    timeout++;
    if (timeout > 10) {
      while (Serial3.available())
        Serial3.read();
      break;
    }
    delay(50);
  }

  for (int i = 0; i < 7; i++)  {
    senseAirResponseBytes[i] = Serial3.read();
  }
}

unsigned long getValue(byte packet[]) {
  int high = packet[3];
  int low = packet[4];
  unsigned long val = high * 256 + low;
  return val;
}

void Serial1SendingDataWorker(unsigned long val) { 
  int sent = (int)val;
  myTransfer.txBuff[0] = highByte(sent);
  Serial.println(myTransfer.txBuff[0]);
  myTransfer.txBuff[1] = lowByte(sent);
  Serial.println(myTransfer.txBuff[1]);
  myTransfer.sendData(sizeof(int));
  delay(100);
  
  for (int i = 0; i < 2; i++) //Clear buffer
  {
    myTransfer.txBuff[i] = 0;
  }
}

Receiving board scetch:

#include "SerialTransfer.h"
SerialTransfer myTransfer;

void setup() {
  Serial.begin(9600);
  while(!Serial); // time to get serial running

  Serial.println(F("Initializing Serial1 (18, 19 pins)..."));
  Serial1.begin(9600);
  while(!Serial1);

  myTransfer.begin(Serial1);
}

void loop() {
  SerialGettingDataWorker();
  
  delay(1000);
}

void SerialGettingDataWorker() {
  if(myTransfer.available())
  {
    Serial.println("New Data");
    for(byte i = 0; i < myTransfer.bytesRead; i++)
      Serial.println(myTransfer.rxBuff[i]);
  }
  else if(myTransfer.status < 0)
  {
    Serial.print("ERROR: ");
    Serial.println(myTransfer.status);
  }
}

Output for sending board:

19:55:56.765 -> Co2 ppm = 724
19:55:56.765 -> 2
19:55:56.765 -> 212
19:55:58.951 -> Co2 ppm = 724
19:55:58.951 -> 2
19:55:58.951 -> 212
19:56:01.102 -> Co2 ppm = 693
19:56:01.102 -> 2
19:56:01.102 -> 181
19:56:03.288 -> Co2 ppm = 682
19:56:03.288 -> 2
19:56:03.288 -> 170
19:56:05.444 -> Co2 ppm = 682
19:56:05.444 -> 2
19:56:05.444 -> 170

Output for receiving board:

19:56:43.278 -> New Data
19:56:43.278 -> 0
19:56:43.278 -> 0
19:56:45.293 -> New Data
19:56:45.293 -> 0
19:56:45.293 -> 0
19:56:47.311 -> New Data
19:56:47.311 -> 0
19:56:47.311 -> 0
19:56:49.328 -> New Data
19:56:49.328 -> 0
19:56:49.328 -> 0
19:56:51.345 -> New Data
19:56:51.345 -> 0
19:56:51.345 -> 0

If to uncomment valCO2 = 746; then output for sending board:

19:58:06.989 -> Co2 ppm = 746
19:58:06.989 -> 2
19:58:06.989 -> 234
19:58:09.161 -> Co2 ppm = 746
19:58:09.161 -> 2
19:58:09.161 -> 234
19:58:11.331 -> Co2 ppm = 746
19:58:11.331 -> 2
19:58:11.331 -> 234

Output for receiving board:

20:00:44.100 -> New Data
20:00:44.100 -> 2
20:00:44.100 -> 234
20:00:46.139 -> New Data
20:00:46.139 -> 2
20:00:46.139 -> 234
20:00:48.152 -> New Data
20:00:48.152 -> 2
20:00:48.152 -> 234

Try using these sketches instead:

TX Arduino:


#include "SerialTransfer.h"




SerialTransfer myTransfer;




byte readCO2[] = {0xFE, 0X44, 0X00, 0X08, 0X02, 0X9F, 0X25};
byte response[] = {0, 0, 0, 0, 0, 0, 0};




void setup()
{
  Serial.begin(9600);
  while(!Serial);

  Serial1.begin(9600);
  while(!Serial1);

  myTransfer.begin(Serial1);

  Serial3.begin(9600);
  while(!Serial3);
}




void loop()
{
  sendRequest(readCO2);
  uint16_t valCO2 = getValue(response);
  
  //valCO2 = 746;
  Serial.print("Co2 ppm = ");
  Serial.println(valCO2);

  sendVal(valCO2);
 
  delay(2000);
}




void sendRequest(byte packet[])
{
  int timeout = 0;

  flushSerial3();
  
  Serial3.write(readCO2, 7);
  delay(50);
  
  while (Serial3.available() < 7)
  {
    timeout++;
    
    if (timeout > 10)
    {
      flushSerial3();
      
      return;
    }

    delay(50);
  }

  for (byte i = 0; i < 7; i++)
    response[i] = Serial3.read();
}




void flushSerial3()
{
  while (Serial3.available())
    Serial3.read();
}




uint16_t getValue(byte packet[])
{
  byte high = packet[3];
  byte low  = packet[4];
  unsigned long val = (high << 8) | low;
  
  return val;
}




void sendVal(uint16_t val)
{
  myTransfer.txObj(val, sizeof(val));
  myTransfer.sendData(sizeof(val));
}

RX Arduino:


#include "SerialTransfer.h"




SerialTransfer myTransfer;




uint16_t valCO2;




void setup()
{
  Serial.begin(9600);
  while(!Serial);

  Serial1.begin(9600);
  while(!Serial1);

  myTransfer.begin(Serial1);
}




void loop()
{
  SerialGettingDataWorker();
}




void SerialGettingDataWorker()
{
  if(myTransfer.available())
  {
    myTransfer.rxObj(valCO2, sizeof(valCo2));
    
    Serial.println("New Data");
    Serial.println(valCO2);
  }
  else if(myTransfer.status < 0)
  {
    Serial.print("ERROR: ");
    Serial.println(myTransfer.status);
  }
}

I've got this for sending board:

19:02:10.498 -> Co2 ppm = 926
19:02:12.554 -> Co2 ppm = 926
19:02:14.639 -> Co2 ppm = 882
19:02:16.692 -> Co2 ppm = 882
19:02:18.750 -> Co2 ppm = 875
19:02:20.834 -> Co2 ppm = 875
19:02:22.887 -> Co2 ppm = 841

And something like this for receiving board:

19:03:32.202 -> New Data
19:03:32.202 -> 4446
19:03:34.286 -> New Data
19:03:34.286 -> 4421
19:03:36.340 -> New Data
19:03:36.340 -> 4421
19:03:38.426 -> New Data
19:03:38.426 -> 4421
19:03:40.480 -> New Data
19:03:40.480 -> 4421
19:03:42.572 -> New Data
19:03:42.572 -> 4384

Also I tried to change back to byte-by-byte sending and get something like this.
For sending board:

19:14:35.867 -> Co2 ppm = 827
19:14:35.867 -> 3
19:14:35.867 -> 59
19:14:37.949 -> Co2 ppm = 808
19:14:37.949 -> 3
19:14:37.949 -> 40
19:14:40.001 -> Co2 ppm = 808
19:14:40.001 -> 3
19:14:40.001 -> 40
19:14:42.086 -> Co2 ppm = 799
19:14:42.086 -> 3
19:14:42.086 -> 31

For receiving board:

19:13:52.631 -> New Data
19:13:52.631 -> 19
19:13:52.631 -> 252
19:13:54.701 -> New Data
19:13:54.701 -> 19
19:13:54.701 -> 250
19:13:56.771 -> New Data
19:13:56.771 -> 19
19:13:56.771 -> 244
19:13:58.844 -> New Data
19:13:58.844 -> 19
19:13:58.844 -> 244
19:14:00.917 -> New Data
19:14:00.917 -> 19
19:14:00.917 -> 239

It looks already as partial success because at least received bytes are different opposite to previous results when they were zeros:) but I still don't see that it's equal to sent numbers.

Ok, found a bug in the TX code. Change getValue()'s definition to this and try again:

uint16_t getValue(byte packet[])
{
  byte high = packet[3];
  byte low  = packet[4];
  uint16_t val = (high << 8) | low; // <--- edited line
  
  return val;
}

Also, try to stay away from sending individual bytes - it's easier and more reliable to transfer entire objects instead.

Power_Broker:
Also, try to stay away from sending individual bytes - it's easier and more reliable to transfer entire objects instead.

I agree, I did that just for testing purpose. Returned it now to sending object.

Also I've changed getValue() but still get:

01:26:32.658 -> 8214
01:26:34.729 -> New Data
01:26:34.729 -> 8212
01:26:36.797 -> New Data
01:26:36.797 -> 8212
01:26:38.864 -> New Data
01:26:38.864 -> 8204
01:26:40.964 -> New Data
01:26:40.964 -> 8204
01:26:43.001 -> New Data
01:26:43.001 -> 8193

As previously if to uncomment hardcode it works perfect - sends and receives 746.

The packets are definitely transferring correctly (else it wouldn't say "New Data"), but it seems the payload bytes within the RX buffer aren't being interpreted correctly.

Just as a sanity check, can you post the code you're testing as of rn?

Exactly the same which you posted.

#include "SerialTransfer.h"

SerialTransfer myTransfer;

uint16_t valCO2;

void setup()
{
  Serial.begin(9600);
  while(!Serial);

  Serial1.begin(9600);
  while(!Serial1);

  myTransfer.begin(Serial1);
}

void loop()
{
  SerialGettingDataWorker();
}

void SerialGettingDataWorker()
{
  if(myTransfer.available())
  {
    myTransfer.rxObj(valCO2, sizeof(valCO2));
    
    Serial.println("New Data");
    Serial.println(valCO2);
  }
  else if(myTransfer.status < 0)
  {
    Serial.print("ERROR: ");
    Serial.println(myTransfer.status);
  }
}

I didn't see internal implementation of rxObj(valCO2, sizeof(valCO2)) though.

Btw, just in case if I find problem in incorrect place. Can it be related to any hardware stuff? For example now my sending board is just Mega2560 with connected Senseair sensor.
Receiving board is Mega2560 with Ethernet shield. I didn't remove it because in real using (not in this testing scetches) I need it. Receiving board gets power from PC via USB. Sending board get's power from receiving board. I can try to power one of the boards via separate source today (I remember that it's needed to connect two boards with common ground in this case). I don't think that it's a reason because initially when I found issue boards had different power suppliers but just in case...

I've tested with separate power suppliers. No success. Received numbers correlate with sent ones but I don't understand why transformation occurs. When I changed code to see which bytes are sent and received I saw something like this transformation table:
Byte 4 was transformed to 11. Byte 3 - to 10, byte 2 - to 9. And it corresponds with numbers which I get when use code with myTransfer.txObj(val, sizeof(val)) instead of byte-by-byte. For example if 3 becomes 10 then likely receiving part of SerialTransfer multiplies it 256 and adds the second byte. It gives something like ~2700 - that's what I observe. I don't know though if the second byte is transformes or not but at least first one definitely yes.

How can it be? It's already nightmare for me.

I didn't see internal implementation of rxObj(valCO2, sizeof(valCO2)) though.

What do you mean by that? The definition of rxObj() is in the header:

	/*
	void SerialTransfer::rxObj(T &val, uint8_t len, uint8_t index)
	 Description:
	 ------------
	  * Reads "len" number of bytes from the receive buffer (rxBuff)
	  starting at the index as specified by the argument "index"
	  into an arbitrary object (byte, int, float, double, struct, etc...)
	 Inputs:
	 -------
	  * T &val - Pointer to the object to be copied into from the
	  receive buffer (rxBuff)
	  * uint8_t len - Number of bytes in the object "val" received
	  * uint8_t index - Starting index of the object within the
	  receive buffer (txBuff)
	 Return:
	 -------
	  * bool - Whether or not the specified index is valid
	*/
	template <typename T>
	bool rxObj(T &val, uint8_t len, uint8_t index=0)
	{
		if (index < (MAX_PACKET_SIZE - len + 1))
		{
			uint8_t* ptr = (uint8_t*)&val;

			for (byte i = index; i < (len + index); i++)
			{
				*ptr = rxBuff[i];
				ptr++;
			}

			return true;
		}

		return false;
	}

The thing is that the packets are getting transferred properly, which leads me to believe the problem more lies in how the packets were formed in the first place.

Since we're using txObj() on the transmitting side, I'm not sure how that would be a problem. Just as a dummy check, if you want to see the logic behind the txObj() function, the definition is below:

	/*
	 void SerialTransfer::txObj(T &val, uint8_t len, uint8_t index)
	 Description:
	 ------------
	  * Stuffs "len" number of bytes of an arbitrary object (byte, int,
	  float, double, struct, etc...) into the transmit buffer (txBuff)
	  starting at the index as specified by the argument "index"
	 Inputs:
	 -------
	  * T &val - Pointer to the object to be copied to the
	  transmit buffer (txBuff)
	  * uint8_t len - Number of bytes of the object "val" to transmit
	  * uint8_t index - Starting index of the object within the
	  transmit buffer (txBuff)
	 Return:
	 -------
	  * bool - Whether or not the specified index is valid
	*/
	template <typename T>
	bool txObj(T &val, uint8_t len, uint8_t index=0)
	{
		if (index < (MAX_PACKET_SIZE - len + 1))
		{
			uint8_t* ptr = (uint8_t*)&val;

			for (byte i = index; i < (len + index); i++)
			{
				txBuff[i] = *ptr;
				ptr++;
			}

			return true;
		}

		return false;
	}

One thing to note is that since Arduinos are generally little-endian, the high and low bytes will look "reversed" when monitoring the outputs of txObj() and rxObj().

Also, please note that since your data is 16-bit, I changed the variable co2val from unsigned long to uint16_t. You only need 2 bytes, so only use 2 bytes.

It looks like I had been finding a program issue for two weeks but it had appeared to be in circuit design. Could you please have a look with your exprienced glance?
I really don't understand why I observe it.

  1. So I bought the second USB cable and connected both boards to PC like on scheme PowerOverUSBForBothMC_bb.png . Everything works perfect!
  2. Then I disconnect one of the board from PC but connect boards with common ground like on PowerOverExternalSupplier_bb.png. What I see at this moment - MC 2's ON diod lights... Obviously board takes power from one of the uart wires. I tried to disconnect wire from 19 (RX) pin and diod turned off. So it looks like MC 1 sends data via own 18 pin, MC 2 gets data via 19 pin, but it also gets power from it...
  3. Then I connect external power from 5V stabilizer like on PowerOverExternalSupplierAndUSB_bb.png.
    And now I see the same problem what I complained initially, i.e. data corruption.

I understand why the first scheme works - both boards use the same power supplier => they have common ground.
I don't understand what happens at the second scheme and the more I totally don't understand why last scheme does not work though boards are connected via common ground.

Largely a riddle to me, too. However, why not connect the 5V pins of the two Arduinos together instead of using an external power supply? Maybe the power supply is bad. Also, have you tried using different serial ports?