[SOLVED] Erraneous transmission at high baud rate for serial communications

Dear all,

On my Arduino Uno, when I set the baud rate at 1,000,000, it worked still great. However the PC is often receiving entangled strings sent from the serial port due to some missing bytes. I guess because it is too high a rate?

For example, I am sending an integer (0-1023) line by line, so it should be:
"51\r\n"
"233\r\n"
"37\r\n"
"166\r\n"

And sometimes, the PC just missed one or two characters. For example, instead of "37\r\n" followed with "11\r\n", it will get "3\r11\r\n" or even "3711\r\n"!

Since this is sensor data, I can throw away some points if I know they are bad points... However there is also no easy & clean way that I can think of to identify these "bad lines".

Any thoughts?

Thanks!

Shawn

yw5aj:
I guess because it is too high a rate?

What is between your Arduino and your computer? If it is just a USB cable then the high baud rate is not the problem.

The culprit is very likely the computer software. I suggest using a known-to-be-reliable terminal application.

Since this is sensor data, I can throw away some points if I know they are bad points... However there is also no easy & clean way that I can think of to identify these "bad lines".

Include a CRC.

Thank you for your input! It is really helpful to know that high baud rate shouldn't be a problem. And yes, I am using a USB cable so the speed doesn't sound like a real problem.

I am using Python's pyserial to read the data. Specifically, I did Serial.println(val) to keep outputting an int, and used ser.readline() to keep reading a line.

Could it be the readline() function? Sorry about being naive on the serial communication, but I do see this line on pyserial documentation:

Note that when the serial port was opened with no timeout, that readline() blocks until it sees a newline (or the specified size is reached)

It shouldn't matter, right? The blocking should mean that no other devices can read the serial port, but arduino should still be able to write to it?

For the reference, below is the complete code:

In Python, reading the data:

import serial
import time

if __name__ == '__main__':
    ser = serial.Serial('COM3', baudrate=1000000)
    data = []
    time0 = time.time()
    while (time.time() - time0 < 5):  # Read data for 5 seconds
        data.append(ser.readline())
    ser.close()

In Arduino:

#define FASTADC 1  // Flag for prescale 16
// Code pasted for modifying ADCSRA
#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

void setup() {
  Serial.begin(1000000);
  #if FASTADC
    // set prescale to 16
    sbi(ADCSRA,ADPS2) ;
    cbi(ADCSRA,ADPS1) ;
    cbi(ADCSRA,ADPS0) ;
  #endif
}

void loop() {
  int val;
  val = analogRead(A0);
  Serial.println(val);
}

Reading data for 5 seconds does not make any sense.

You need to have start and end markers so the Python program knows when the data starts and end.

Look at this Python - Arduino demo (which is sticky at the top of this section ! )

...R

Robin2:
Reading data for 5 seconds does not make any sense.

You need to have start and end markers so the Python program knows when the data starts and end.

Look at this Python - Arduino demo (which is sticky at the top of this section ! )

...R

Thank you Robin for your response! I did read your code before you post, but I thought that was a different kind of data being read. Mine was continuous data streaming from the sensor, with delimiters ("\r\n"). So reading the data for 5 seconds is a task that I need to do, without specifying a start and an end - or, the start and the end is the delimiters.

But I do think I should probably take a closer look to see whether there are concepts that I can convert to do the continuously reading task. Thanks again :slight_smile:

The first line is very likely sheared. Do you throw out the first line in your post processing?

Yes I did throw the first data point away. The evils are in the middle. For example, if I plot my data, it look like this:

Where those spikes are incomplete data force-converted to int.

Actually - it might be the high baud rate! With more careful search, I found these posts:

http://forum.arduino.cc/index.php?topic=131872.0

People mention that at high baud rates, errors occur.

More interestingly!!

If I just send sine wave data generated, the 500,000 baud rate works OK. But, if I read from the analog pin, using analogRead(A0), then errors suddenly start to occur!

Obviously, it is analogRead(A0) that is messing with the serial communication. Any ideas?

Shawn

I have an Arduino Uno Rev 3.

I have a force sensitive resistor as my sensor. Current baud rate: 1,000,000. It works now:

  1. Perfectly if I am sending generated sine wave (a very small an amplitude with only peak magnitude of 1) + sensor output;
  2. Erroneous if I am sending sensor readout alone.

The "sine wave + sensor" code and plot:

#define FASTADC 1  // Flag for prescale 16
// Code pasted for modifying ADCSRA
#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

void setup() {
  Serial.begin(1000000);
  #if FASTADC
    // set prescale to 16
    sbi(ADCSRA,ADPS2) ;
    cbi(ADCSRA,ADPS1) ;
    cbi(ADCSRA,ADPS0) ;
  #endif
}

void loop() {
  int val;
  val = analogRead(A0);
  val += int(512 * (sin(float(millis()) * 0.01) + 1)) * 0.001;
  Serial.println(val);
}

For the code of sensor alone, just comment out the "val += int(...." line.

The weird spikes are actually caused by missing bytes (characters here). For example, instead of "37\r\n" followed with "11\r\n", it will get "3\r11\r\n" or even "3711\r\n"!

The Python code to read the data are the same:

import serial
import time


if __name__ == '__main__':
    ser = serial.Serial('COM3', baudrate=1000000)
    time0 = time.time()
    f = open('data.csv', 'w')
    ser.readline()
    while (time.time() - time0 < 10):
        try:
            data = int(ser.readline())
            current_time = time.time() - time0
            f.write('%.8f, %d\n' % (current_time, data))
            f.flush()
        except ValueError:
            pass
    ser.close()
    f.close()


    # %% Plot data
    import matplotlib.pyplot as plt
    import numpy as np
    time_array, data_array = np.loadtxt('data.csv', delimiter=',').T
    plt.plot(time_array, data_array)
    plt.ylim(-100, 1500)
    plt.xlabel('Time (s)')
    plt.ylabel('Data (0-1023)')
    plt.tight_layout()
    plt.savefig('data_sensor_alone.png')

yw5aj:
Actually - it might be the high baud rate! With more careful search, I found these posts:

USB Serial Baud Rates? ... or other faster USB communication methods? - Arduino Due - Arduino Forum

People mention that at high baud rates, errors occur.

More interestingly!!

If I just send sine wave data generated, the 500,000 baud rate works OK. But, if I read from the analog pin, using analogRead(A0), then errors suddenly start to occur!

Obviously, it is analogRead(A0) that is messing with the serial communication. Any ideas?

Shawn

I just realized that this is indeed a different issue than the OP. I'll open a new thread then!

Here it is: http://forum.arduino.cc/index.php?topic=316446.0

yw5aj:
The weird spikes are actually caused by missing bytes (characters here).

Why does the generated sine wave not have spikes?

The "wave" you saw is my finger pressing on it. The generated sine wave, as you can see in the code, is only having peak magnitude of 1, which is trivial compared to the sensor input.

Good question - sorry that I did not made it more clear in the OP. I will edit it now.

yw5aj:
I just realized that this is indeed a different issue than the OP.

By your own admission, the topics are the same (dropped data).

Threads merged.

I found an easier way to repeat the problem.

void setup() {
  Serial.begin(2000000);
}

void loop() {
  Serial.println(234);
}

Keep sampling with your favorite tool for 10,000 data points - there will be spikes.

Shawn

Hi Coding Badly,

I feel that this post deviated far enough from the OP, as I have made progresses to narrow down the source of the problem. People will get lost if I keep posting here. Could you please help review whether the following post is OK to create a new one? It would be OK to say no too and I really appreciate your work in help organizing this forum.

Subject: Sending constant int from Arduino to PC causes error

I am using Arduino Uno Rev 3, and according to this link we can achieve 2000000 baud rate.

This problem can be easily repeated by the following code:

void setup() {
  Serial.begin(2000000);
}

void loop() {
  int val = 234;
  Serial.println(val);
}

I am using the serial-USB cable. To read the data, I used Python pyserial (< 20 lines of code including empty lines and import lines):

import serial
import matplotlib.pyplot as plt

LEN = 10000
data = []
ser = serial.Serial('COM3', baudrate=2000000)
for i in range(LEN):
    try:
        data.append(int(ser.readline()))
    except:
        pass
ser.close()

# %% Plot data
plt.figure(figsize=(6, 4))
plt.plot(data)
plt.savefig('data.png')

Here is what I got:

yw5aj:
Could you please help review whether the following post is OK to create a new one?

Really? Rather than helping you with your problem you want me to waste my time justifying my moderation decisions? I will pass.

Don't take me wrong please! Again, as I said in the post above, I really appreciate your time and help in organizing the forum threads. Instead of trying to fight, I was thinking whether that would be a better way to discuss this issue. I will not post a new thread then.

Please don't misunderstand me, who truly appreciate all your effort!

Shawn

I cannot get it to fail (excess 234s trimmed)...

C:\ >C:\Python34\python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import serial
>>>
>>> LEN = 10000
>>> data = []
>>> ser = serial.Serial('COM3', baudrate=2000000)
>>> for i in range(LEN):
...     try:
...         data.append(int(ser.readline()))
...     except:
...         pass
...
>>> ser.close()
>>>
>>> data
[234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
....
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234]
>>>

...which brings us full circle to Reply #1. Your Python script is not removing data from the serial buffer fast enough. You need some combination of: faster PC, better kernel driver, faster Python run-time, different computer-side programming language, framing, error detection.