Strange Serial behavior on Uno

I am using C#/Unity on the host and doing echo testing on a Teensy, Mega, and Uno - to verify my code. On the microcontrollers I am running a sketch based on Nick Gammon’s page here:

http://www.gammon.com.au/serial

I send an identical message with a fixed delay between sends - comparing the number sent from the host to the number echoed back by the micro. The results with the Teensy are as expected - at any delay, with any size message, the tests always pass (e.g. 10,000 messages out and back with no missing characters).

With the Uno and Mega, the results are more mixed. With very short messages of one or two bytes, I can send at any interval I want and results are good. However, with longer messages of 20 bytes of most messages. The part that throws me off though, is the more infrequent the messages the more bytes I lose(!). For example, if I send each message as soon as the previous send returns, about 4% of the messages are impaired in some way. BUT - if I send each message spaced 1 sec apart, the failure rate is over 80%.

Here is the sketch:

// how much serial data we expect before a newline
const int MAX_INPUT = 50;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  while (!Serial) {
    // wait for serial port to connect. Needed for native USB
  }
}

void loop() {
  // just keep reading serial and processing the bytes
  readSerial();
}

void readSerial()
{
  while (Serial.available ()) {
    processIncomingByte (Serial.read ());
  }
}

void processIncomingByte (const byte inByte)
{
  static char inLine[MAX_INPUT];
  static int inPos = 0;

  switch (inByte)
  {    
    case '\n':   // end of text
      inLine [inPos] = 0;  // terminating null byte

      // terminator reached! process input_line here ...
      processMessage(inLine);

      // reset buffer for next time
      inPos = 0;
      break;

    case '\r':   // discard carriage return  
     break;

    default:
      // keep adding if not full ... allow for terminating null byte
      if (inPos < (MAX_INPUT - 1))
        inLine [inPos++] = inByte;
      break;

  }  // end of switch

} // end of processIncomingByte

void processMessage(char* buf)
{
  // processing of serial command goes here
  // in this sketch, just echo the message
  writeSerial(buf);
}

void writeSerial(char* buf)
{
  Serial.println(buf);
}

And here is the C# serial code:

using UnityEngine;
using System.Collections;
using System.IO.Ports;

public class Arduino : MonoBehaviour
{

    [SerializeField] private string port;
    [SerializeField] private int baudrate;

    SerialPort stream;

    public delegate void MessageReceived(string message);
    public static event MessageReceived OnMessageReceived;


    // runs on startup - init Serial
    void Start()
    {
        try
        {
            stream = new SerialPort(@"\\.\" + port, baudrate);
            stream.ReadTimeout = 1;
            stream.Open();
        }
        catch (System.Exception e)
        {
            print(e.Message);
        }
    }

    // runs every frame and checks for incoming serial data
    void Update()
    {
        try
        {
            string m = stream.ReadLine();
            OnMessageReceived(m);
        }
        catch (System.TimeoutException)
        {
        }
        catch (System.Exception e)
        {
            print(e.Message);
        }
    }


    //  takes string and writes to serial
    public void SendSerialMessage(string m)
    {
        try
        {
            stream.WriteLine(m);
        }
        catch (System.Exception e)
        {
            print(e.Message);
        }
    }
}

Which I call from another script for testing, like this:

        int idx = 0;
        while (idx < 10000)
        {
            arduino.SendSerialMessage(message);
            yield return new WaitForSeconds(sendPause); // this pauses before sending again (longer pause means more lost data on the Uno)
            idx++;
        }

I don't use C# so I don't know if your code is closing and opening the serial connection. Doing that cause an Uno and Mega to reset which takes a few seconds and might cause the symptoms you see. It may be that the Teensy behaves like a Leonardo or Micro and does not reset when the serial connection is opened by the PC software.

Make sure that your PC program opens the serial connection, allows time for the Arduino to reset, and then keeps the connection open until it is completely finished with the Arduino.

...R

It is not clear what you are sending, what you are receiving, or what you think you are loosing. Some more details are clearly in order.

you are probaby overrunning serial buffers with long messages
you could use handshaking flow control either hardware or software such as XON/XOFF

PaulS: It is not clear what you are sending, what you are receiving, or what you think you are loosing. Some more details are clearly in order.

Here are two samples I just tested:

Passing stream.Writeline() the parameter "t", there were 10,000 sent and 10,000 returned.

Passing stream.Writeline() "much longer test message", 10,000 sent and 9,919 were returned intact. 39 were returned with an unexpected newline or CR at the front, 1 returned as "st message" and 1 returned blank, with 40 missing.

One new thing I noticed this time around - If I send as fast as possible, the failures all happen early in the test (within the first 500 sends) and then the remaining messages all return intact. If buffers were overrun, would we expect the results to get worse over time - or at least no better?

I will implement a version that uses a C# function which sends single bytes now and check those results. Even if it works though - still curious to hear thoughts on what is happening here.

Edit: should also mention I am not getting exceptions on the C# side...

have you enabled the C# SerialPort.ErrorReceived Event? https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.errorreceived(v=vs.110).aspx

the SerialErrorReceivedEventArgs parameter will give you the SerialError Enumeration will may indicate overrun errors https://msdn.microsoft.com/en-us/library/system.io.ports.serialerror(v=vs.110).aspx

interesting that your tests run Ok on a Teensy which is a faster microcontroller than the Uno or Mega

horace:
have you enabled the C# SerialPort.ErrorReceived Event?

Good thought, I tried it at your suggestion, but got no events. It is possible that Mono (which Unity uses) doesn’t support that event - the .NET SerialPort class is not fully implemented in Mono: HowToSystemIOPorts | Mono

I found a solution that works very well though! Reading one byte at a time solved the dropped bytes using the UNO, but was too slow for the Teensy unless I risked blocking. In the end I used SerialPort.Read (Byte, Int32, Int32) to read in larger chunks.

Here is the final version of the methods I used for reading and parsing serial from the Arduino and some of the declarations:

    private const int bufSize = 2048;   // buffer for storing all incoming bytes and parsing them
    private const byte CR = 0xD;        // carriage return byte
    private const byte LF = 0xA;        // line feed (newline) byte - write Arduino sketch to use Println or use LF as line terminator

    private byte[] buf = new byte[bufSize];                 // buffer for writing incoming serial
    private byte[] messageBuilder = new byte[bufSize * 2];  // buffer for storing incoming serial during parsing
    private int idx = 0;      

    private IEnumerator ReadSerial()
    {
        while (true)
        {
            // if port was closed after disconnect and autoReconnect
            if (!stream.IsOpen && autoReconnect)
            {
                do
                {
                    Connect();
                    yield return new WaitForSeconds(1f);
                } while (!stream.IsOpen);
            }
            try
            {
                // read up to bufSize bytes from the Serial buffer
                int readCount = stream.Read(buf, 0, bufSize);
                // send buffer on for parsing out the messages
                ParseBuffer(buf, readCount);
            }
            catch (TimeoutException)
            {
            }
            catch (System.IO.IOException e)
            {
                // Reader probably unplugged.  Close so that IsOpen() will read false
                print(Prepend(e.Message));
                print(Prepend("Closing connection."));
                stream.Close();
            }
            catch (Exception e)
            {
                print(Prepend(e.ToString()));
            }
            yield return new WaitForSeconds(0.0f);
        }
    }

    private void ParseBuffer(byte[] buf, int len)
    {
        // parse up to len bytes (should be the number read into buf)
        for (int i = 0; i < len; i++)
        {
            byte inByte = buf[i];
            switch (inByte)
            {
                // ignore carriage return
                case CR:
                    break;
                // message end is line-feed, reset idx for next message
                case LF:
                    idx = 0;
                    HandleMessage(messageBuilder);
                    break;
                // anything else is part of message, write it to next position
                default:
                    messageBuilder[idx++] = inByte;
                    break;
            }
        }
    }

I wish I had a more satisfying answer to contribute as to why ReadLine() was missing bytes on the Uno, but this method has been great. I tested with different length messages on the Uno and Teensy and with over 50,000 messages of different lengths, none were lost or missing bytes.