Corrupted serial data

Hello.

I have a bit of a complicated affair going on with comms between a Python routine, a Mega2560 and an Uno.

My Mega is sending data on Serial3 as stated below: (code is huge, so just lifted this bit)

SendtoUNO is called whenever the data changes (that works fine).

//--Send data from the Mega to the UNO so it can be passed on to the Python routine later

char Assembled_data[20];                                                         // Data to send to the Uno
char Number[5];                                                                      // To assist with data conversion to send to the Uno



void SendtoUNO(){
  
  strcpy(Assembled_data, "");                                                   // Clear the data
  strcat(Assembled_data, "<");                                                  // Assemble the data
  sprintf(Number, "%d", int(Ammo));  strcat(Assembled_data, Number);
  strcat(Assembled_data, ",");
  sprintf(Number, "%d", Detection_mode);  strcat(Assembled_data, Number);
  strcat(Assembled_data, ",");
  sprintf(Number, "%d", Firing);  strcat(Assembled_data, Number);
  strcat(Assembled_data, ">");

  Serial3.println(Assembled_data);

}

This is then received on the UNO as:

const byte buffSize2560 = 40;
char inputBuffer2560[buffSize2560];
byte bytesRecvd2560 = 0;
boolean readInProgress2560 = false;
boolean newDataFrom2560 = false;
const char startMarker = '<';
const char endMarker = '>';



void loop() {

  if(LinkSerial.available() > 0) {

    char z = LinkSerial.read();               // read char from serial
      
    if (z == endMarker) {                     // look for end marker
      readInProgress2560 = false;             // if found, set read in progress true (will stop adding new byte to buffer)
      inputBuffer2560[bytesRecvd2560] = 0;    // clear input buffer
      processData2560();                      // process data in buffer
    }
    
    if (readInProgress2560) {
      inputBuffer2560[bytesRecvd2560] = z;    // populate input buffer with bytes
      bytesRecvd2560 ++;                      // increment index
      if (bytesRecvd2560 == buffSize2560) {   // when buffer is full
        bytesRecvd2560 = buffSize2560 - 1;    // keep space for end marker
      }
    }

    if (z == startMarker) {                   // look for start maker
      bytesRecvd2560 = 0;                     // if found, set byte received to 0
      readInProgress2560 = true;              // set read in progress true
    }
  }


}

//--------------------------------------------------------

void processData2560()                            // Convert data from 2560p to useable variables
{
  char * strtokIndx2560;                          // this is used by strtok() as an index

   strtokIndx2560 = strtok(inputBuffer2560,",");  // get the first part                                
   Ammo = atoi(strtokIndx2560);                   // convert this part to an integer                    

   strtokIndx2560 = strtok(NULL,",");             // get the second part(this continues where the previous call left off)
   Detection_mode_gun = atoi(strtokIndx2560);     // convert this part to an integer                   
   strtokIndx2560 = strtok(NULL, ",");            // get the third part
   Firing = atoi(strtokIndx2560);                 // convert this part to an integer (string to int)     

}

The data is arriving fine sometimes, but corrupted probably 1 in 5 reads.
Have not been able to find the reason.

Can somebody confirm whether my method is sound?

Your Uno code looks very similar to the code in the 3rd example in Serial Input Basics and if that's where you got your inspiration I wonder why you did not use my code without any changes - apart from the size of the buffer.

...R

Your SenttoUNO code can be reduced to

void SendtoUNO(){
  sprintf(Assembled_data, "<%d,%d,%d>", int(Ammo), Detection_mode, Firing);
  Serial3.println(Assembled_data);
}

To be honest, I have no idea where I found the original code.. I lifted it from a previous project.
So, yes.... probably your code that has got tinkered around with, or a copy of your code via some other website/source.

The problem is the receiving.

My loop is doing other things when the data arrives and its getting missed.
I thought the Serial.available would have flagged there was data to read on it's next loop around, but that doesn't appear to be happening.

If I repeat the 'check for serial data' multiple times in the loop, then results are better.... but still not perfect.

Is there any way to trigger a 'read the data' sub routine the instant any data arrives?

Not sure why it's not being kept to be read next loop around. The buffer is plenty big enough

OK.

I added a checksum, and slowed the data from 115200 to 9600. Seems better now.

It would be much faster and reliable to interface Python with your Arduino using the compatible libraries: pySerialTransfer and SerialTransfer.h.

pySerialTransfer is pip-installable and cross-platform compatible. SerialTransfer.h runs on the Arduino platform and can be installed through the Arduino IDE's Libraries Manager.

Both of these libraries have highly efficient and robust packetizing/parsing algorithms with easy to use APIs.

Example Python Script:

import time
from pySerialTransfer import pySerialTransfer as txfer


if __name__ == '__main__':
    try:
        link = txfer.SerialTransfer('COM17')
        
        link.open()
        time.sleep(2) # allow some time for the Arduino to completely reset
        
        while True:
            send_size = 0
            
            ###################################################################
            # Send a list
            ###################################################################
            list_ = [1, 3]
            list_size = link.tx_obj(list_)
            send_size += list_size
            
            ###################################################################
            # Send a string
            ###################################################################
            str_ = 'hello'
            str_size = link.tx_obj(str_, send_size) - send_size
            send_size += str_size
            
            ###################################################################
            # Send a float
            ###################################################################
            float_ = 5.234
            float_size = link.tx_obj(float_, send_size) - send_size
            send_size += float_size
            
            ###################################################################
            # Transmit all the data to send in a single packet
            ###################################################################
            link.send(send_size)
            
            ###################################################################
            # Wait for a response and report any errors while receiving packets
            ###################################################################
            while not link.available():
                if link.status < 0:
                    if link.status == -1:
                        print('ERROR: CRC_ERROR')
                    elif link.status == -2:
                        print('ERROR: PAYLOAD_ERROR')
                    elif link.status == -3:
                        print('ERROR: STOP_BYTE_ERROR')
            
            ###################################################################
            # Parse response list
            ###################################################################
            rec_list_  = link.rx_obj(obj_type=type(list_),
                                     obj_byte_size=list_size,
                                     list_format='i')
            
            ###################################################################
            # Parse response string
            ###################################################################
            rec_str_   = link.rx_obj(obj_type=type(str_),
                                     obj_byte_size=str_size,
                                     start_pos=list_size)
            
            ###################################################################
            # Parse response float
            ###################################################################
            rec_float_ = link.rx_obj(obj_type=type(float_),
                                     obj_byte_size=float_size,
                                     start_pos=(list_size + str_size))
            
            ###################################################################
            # Display the received data
            ###################################################################
            print('SENT: {} {} {}'.format(list_, str_, float_))
            print('RCVD: {} {} {}'.format(rec_list_, rec_str_, rec_float_))
            print(' ')
    
    except KeyboardInterrupt:
        link.close()
    
    except:
        import traceback
        traceback.print_exc()
        
        link.close()

Example Arduino Sketch:

#include "SerialTransfer.h"


SerialTransfer myTransfer;


void setup()
{
  Serial.begin(115200);
  myTransfer.begin(Serial);
}


void loop()
{
  if(myTransfer.available())
  {
    // send all received data back to Python
    for(uint16_t i=0; i < myTransfer.bytesRead; i++)
      myTransfer.txBuff[i] = myTransfer.rxBuff[i];
    
    myTransfer.sendData(myTransfer.bytesRead);
  }
}

On the Arduino side, you can use myTransfer.txObj() and myTransfer.rxObj() to copy values to the library's RX buffer and parse multi-byte variables out of the library's TX buffer.

For theory behind robust serial communication, check out the tutorials Serial Input Basics and Serial Input Advanced.

Comms between the Arduino and Python are fine. No glitches there at all.
It was the comms between the Uno and the Mega that were giving me issues.

phoneystark2020:
To be honest, I have no idea where I found the original code.. I lifted it from a previous project.
So, yes.... probably your code that has got tinkered around with, or a copy of your code via some other website/source.

Either use @Power_Broker's library or use the code that is in my Tutorial.

Or it might be even simpler just to use I2C for communication between the Arduinos. See this I2C Tutorial

...R

Thanks Robin
The Arduinos are a bit too far apart for reliable I2C. Had trouble there before.
The checksum and lower baud seem to have helped

phoneystark2020:
The Arduinos are a bit too far apart for reliable I2C.

I'm curious, do you mind telling us what the distance is?

...R

About 500mm. The I2C is already being used for sensors and an MP3 module. I found it to be quite susceptible to supply and motor noise.

The supply is well suppressed. Main reason for not changing it to I2C is all the re-programmming.
I know it's probably not much for some people, but I don't want to break what I have working.

This MegaPoints YouTube Video shows I2C working over 100m of cable. The MegaPoints boards use Atmega microprocessors.

...R