Unreliable Serial Communication with PC

Hi,

I am pass data from a sensor to a Python program on PC. I am using the USB port in Arduino UNO. It generally works, but after getting 100,000 or so data points, typically more that half an hour, I may get a garbage point. Once that happens, the chance of getting further garbage is much higher.

The code is something like this:

...

Serial.begin(9600);
...

Serial.print(intVal);
Serial.print(',');
Serial.println(floatVal);

On the PC side, I open the port, ser, then wait for the data to come in:

.....

if ser.inWaiting() == 0:
continue

buf = ""

while True:

x=ser.read()

if x == b'\r' or x==b'\n':
pass
else:
buf = buf + x.decode('utf-8')

if x == b'\r':

print("Do something that takes a while.....")

buf = ""

I try to isolate the exact code but the re-produced code hasn't show the same problem yet. I just want to know if this is a well known problem and what I can do to solve this problem.

Yes, it is well known and is in the code you didn't show.

Paul

Glad to know the code I did show is OK, lol.

pax_electronica:
Glad to know the code I did show is OK, lol.

I meant there is something in your code that causing the problem. Computer programs ALWAYS do exactly what you programed them to do. Your task is to find what you did to cause the problem. By not showing the program with the problem, no one else can even guess what you did.

Paul

Paul_KD7HB:
I meant there is something in your code that causing the problem. Computer programs ALWAYS do exactly what you programed them to do. Your task is to find what you did to cause the problem. By not showing the program with the problem, no one else can even guess what you did.

Paul

My simplified test program ran for hours without showing the problem and posting the full program would be distracting. So my question is general in nature, about whether it is "well known" that serial print sends garbage occasionally and what the solution might be. I tried in two computers both running Win 7 both showing the problem. I found a few long, wandering conversations on this topic. They seem to confirm the problem but offer no solution on Windows. I haven't tried Linux.

pax_electronica:
So my question is general in nature, about whether it is "well known" that serial print sends garbage occasionally and what the solution might be

In general it works very reliably for an unlimited period of time.

For the specifics we need to see the two programs.

...R

"I haven't been able to find a piece of example code (both the Python and the Arduino stuff) that can reliably send and receive text and/or binary data between a PC and an Arduino. So I have written a pair of demo programs - one in Python and one for the Arduino - in the hope that they might be useful to newcomers."

I recall seeing this some time ago. That is at least part of the reason I asked this question in the first place. I tested said programs and found it a bit on the slow side for my application. I wonder what is the reason for going to such length for a seeming simple problem.

Now I know, but why?

pax_electronica:
I recall seeing this some time ago.

Without seeing the source of your quotation or the programs I can’t comment.

If the programs work reliably it suggests that their author previously had some unreliable code - which (I presume) is unavailable for comparison.

…R

For the Python code, you can use the pip-installable package pySerialTransfer to packetize your data and send it to an Arduino on a serial port. It can also parse data packets from the Arduino.

Example Python script:

from pySerialTransfer import pySerialTransfer as txfer

if __name__ == '__main__':
    try:
        link = txfer.SerialTransfer('COM13')
    
        link.txBuff[0] = 'h'
        link.txBuff[1] = 'i'
        link.txBuff[2] = '\n'
        
        link.send(3)
        
        while not link.available():
            if link.status < 0:
                print('ERROR: {}'.format(link.status))
            
        print('Response received:')
        
        response = ''
        for index in range(link.bytesRead):
            response += chr(link.rxBuff[index])
        
        print(response)
        link.close()
        
    except KeyboardInterrupt:
        link.close()

On the Arduino side, you can use the compatible Arduino library SerialTransfer.h (installable through the Arduino IDE’s Libraries Manager) to parse the data from Python and send data back if necessary.

Example Arduino Code:

#include "SerialTransfer.h"
#include <SoftwareSerial.h>


SoftwareSerial mySerial(2, 3); // RX, TX
SerialTransfer myTransfer;


void setup()
{
  Serial.begin(115200);
  mySerial.begin(9600);
  myTransfer.begin(mySerial);
}

void loop()
{
  myTransfer.txBuff[0] = 'h';
  myTransfer.txBuff[1] = 'i';
  myTransfer.txBuff[2] = '\n';
 
  myTransfer.sendData(3);
  delay(100);

  if(myTransfer.available())
  {
    Serial.println("New Data");
    for(byte i = 0; i < myTransfer.bytesRead; i++)
      Serial.write(myTransfer.rxBuff[i]);
    Serial.println();
  }
  else if(myTransfer.status < 0)
  {
    Serial.print("ERROR: ");
    Serial.println(myTransfer.status);
  }
}

It turns out that the problem is not with Arduino sending junk. It is the overflow of the USB serial buffer on the PC side.

In the code fragment above, the Arduino keeps sending data and the Python code on PC does the reading and going off to do something. The work slows down the reading and the incoming data is then placed into the USB buffer, which eventually overflows.

I made the Python code ran a bit faster and the problem went away. It also explained why my simplified test code above was not able to reproduce the problem: it runs too fast.

The pySerial package has a set_input_buffer(tx=...,rx=...) but somehow it has no effect on my machine.

The PC input buffer has a max value and cannot save your program from inefficient code. You need to make sure your Python is nimble enough to handle Serial data once it arrives instead of trying to increase the input buffer.

If you use pySerialTransfer, you can constantly check for, and parse, a new packet from the Arduino throughout the "work" that takes a while within your Python script.

Either that or make sure the Arduino doesn't send as much data as it does now.

Yes, it looks like a good option. But for now, simply speeding up the Python read works.

It was a bit surprising that no exceptions were thrown for buffer overflow or for doing nothing with set_input_buffer().

pax_electronica:
Yes, it looks like a good option. But for now, simply speeding up the Python read works.

What do you mean? I don't recall you saying anything about "speeding up the Python read"? Does that mean you've fixed your problem?

Yes. The problem is gone after I sped up the Python program.

Serial comms can be very reliable.
1)
Use acknowledgements; Arduino sends something and waits for confirmation that it was received correctly or incorrectly.
2)
Add a checksum or crc; PC will check and let Arduino know.
3)
As you're sending text, implement a software handshake with XON/XOFF. The.PC will tell the Arduino to stop sending when the buffer fills up and tell the Arduino to start sending again when there is space in the buffer. I've used it the other way (Arduino tells PC to stop or start), never tried PC telling Arduino to stop/start.