Arduino Forum

Using Arduino => Interfacing w/ Software on the Computer => Topic started by: yw5aj on Apr 16, 2015, 03:10 am

Title: [SOLVED] Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 16, 2015, 03:10 am
Dear all,

On my Arduino Uno, when I set the baud rate at 1,000,000, it worked still great. However the PC is often receiving entangled strings sent from the serial port due to some missing bytes. I guess because it is too high a rate?

For example, I am sending an integer (0-1023) line by line, so it should be:
"51\r\n"
"233\r\n"
"37\r\n"
"166\r\n"

And sometimes, the PC just missed one or two characters. For example, instead of "37\r\n" followed with "11\r\n", it will get "3\r11\r\n" or even "3711\r\n"!

Since this is sensor data, I can throw away some points if I know they are bad points... However there is also no easy & clean way that I can think of to identify these "bad lines".

Any thoughts?

Thanks!

Shawn
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 16, 2015, 05:08 am
I guess because it is too high a rate?
What is between your Arduino and your computer?  If it is just a USB cable then the high baud rate is not the problem.

The culprit is very likely the computer software.  I suggest using a known-to-be-reliable terminal application.

Quote
Since this is sensor data, I can throw away some points if I know they are bad points... However there is also no easy & clean way that I can think of to identify these "bad lines".
Include a CRC.

Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 16, 2015, 06:23 am
Thank you for your input! It is really helpful to know that high baud rate shouldn't be a problem. And yes, I am using a USB cable so the speed doesn't sound like a real problem.

I am using Python's pyserial to read the data. Specifically, I did Serial.println(val) to keep outputting an int, and used ser.readline() to keep reading a line.

Could it be the readline() function? Sorry about being naive on the serial communication, but I do see this line on pyserial documentation:

Quote
Note that when the serial port was opened with no timeout, that readline() blocks until it sees a newline (or the specified size is reached)
It shouldn't matter, right? The blocking should mean that no other devices can read the serial port, but arduino should still be able to write to it?



For the reference, below is the complete code:

In Python, reading the data:
Code: [Select]

import serial
import time

if __name__ == '__main__':
    ser = serial.Serial('COM3', baudrate=1000000)
    data = []
    time0 = time.time()
    while (time.time() - time0 < 5):  # Read data for 5 seconds
        data.append(ser.readline())
    ser.close()


In Arduino:
Code: [Select]

#define FASTADC 1  // Flag for prescale 16
// Code pasted for modifying ADCSRA
#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

void setup() {
  Serial.begin(1000000);
  #if FASTADC
    // set prescale to 16
    sbi(ADCSRA,ADPS2) ;
    cbi(ADCSRA,ADPS1) ;
    cbi(ADCSRA,ADPS0) ;
  #endif
}

void loop() {
  int val;
  val = analogRead(A0);
  Serial.println(val);
}
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Robin2 on Apr 16, 2015, 08:51 am
Reading data for 5 seconds does not make any sense.

You need to have start and end markers so the Python program knows when the data starts and end.

Look at this Python - Arduino demo (http://forum.arduino.cc/index.php?topic=225329.msg1810764#msg1810764) (which is sticky at the top of this section ! )

...R
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 16, 2015, 06:22 pm
Reading data for 5 seconds does not make any sense.

You need to have start and end markers so the Python program knows when the data starts and end.

Look at this Python - Arduino demo (http://forum.arduino.cc/index.php?topic=225329.msg1810764#msg1810764) (which is sticky at the top of this section ! )

...R
Thank you Robin for your response! I did read your code before you post, but I thought that was a different kind of data being read. Mine was continuous data streaming from the sensor, with delimiters ("\r\n"). So reading the data for 5 seconds is a task that I need to do, without specifying a start and an end - or, the start and the end is the delimiters.

But I do think I should probably take a closer look to see whether there are concepts that I can convert to do the continuously reading task. Thanks again :)
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 16, 2015, 07:48 pm

The first line is very likely sheared.  Do you throw out the first line in your post processing?

Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 16, 2015, 09:05 pm
Yes I did throw the first data point away. The evils are in the middle. For example, if I plot my data, it look like this:

(https://github.com/yw5aj/arduino/blob/master/daq/temp.png?raw=true)

Where those spikes are incomplete data force-converted to int.
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 16, 2015, 10:36 pm
Actually - it might be the high baud rate! With more careful search, I found these posts:

http://forum.arduino.cc/index.php?topic=131872.0

People mention that at high baud rates, errors occur.
 

**More interestingly!!**

If I just send sine wave data generated, the 500,000 baud rate works OK. But, if I read from the analog pin, using analogRead(A0), then errors suddenly start to occur!

Obviously, it is analogRead(A0) that is messing with the serial communication. Any ideas?


Shawn
Title: Strange behavior: sending wrong signal with sensor data alone
Post by: yw5aj on Apr 17, 2015, 02:59 am
I have an Arduino Uno Rev 3.

I have a force sensitive resistor as my sensor. Current baud rate: 1,000,000. It works now:

1) Perfectly if I am sending generated sine wave (a very small an amplitude with only peak magnitude of 1) + sensor output;
2) Erroneous if I am sending sensor readout alone.

The "sine wave + sensor" code and plot:

Code: [Select]

#define FASTADC 1  // Flag for prescale 16
// Code pasted for modifying ADCSRA
#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

void setup() {
  Serial.begin(1000000);
  #if FASTADC
    // set prescale to 16
    sbi(ADCSRA,ADPS2) ;
    cbi(ADCSRA,ADPS1) ;
    cbi(ADCSRA,ADPS0) ;
  #endif
}

void loop() {
  int val;
  val = analogRead(A0);
  val += int(512 * (sin(float(millis()) * 0.01) + 1)) * 0.001;
  Serial.println(val);
}


(https://github.com/yw5aj/arduino/blob/master/daq/data_sine_and_sensor.png?raw=true)

For the code of sensor alone, just comment out the "val += int(...." line.

(https://github.com/yw5aj/arduino/blob/master/daq/data_sensor_alone.png?raw=true)

The weird spikes are actually caused by missing bytes (characters here). For example, instead of "37\r\n" followed with "11\r\n", it will get "3\r11\r\n" or even "3711\r\n"!

The Python code to read the data are the same:

Code: [Select]

import serial
import time


if __name__ == '__main__':
    ser = serial.Serial('COM3', baudrate=1000000)
    time0 = time.time()
    f = open('data.csv', 'w')
    ser.readline()
    while (time.time() - time0 < 10):
        try:
            data = int(ser.readline())
            current_time = time.time() - time0
            f.write('%.8f, %d\n' % (current_time, data))
            f.flush()
        except ValueError:
            pass
    ser.close()
    f.close()


    # %% Plot data
    import matplotlib.pyplot as plt
    import numpy as np
    time_array, data_array = np.loadtxt('data.csv', delimiter=',').T
    plt.plot(time_array, data_array)
    plt.ylim(-100, 1500)
    plt.xlabel('Time (s)')
    plt.ylabel('Data (0-1023)')
    plt.tight_layout()
    plt.savefig('data_sensor_alone.png')
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 17, 2015, 03:00 am
Actually - it might be the high baud rate! With more careful search, I found these posts:

http://forum.arduino.cc/index.php?topic=131872.0

People mention that at high baud rates, errors occur.
 

**More interestingly!!**

If I just send sine wave data generated, the 500,000 baud rate works OK. But, if I read from the analog pin, using analogRead(A0), then errors suddenly start to occur!

Obviously, it is analogRead(A0) that is messing with the serial communication. Any ideas?


Shawn
I just realized that this is indeed a different issue than the OP. I'll open a new thread then!

Here it is:  http://forum.arduino.cc/index.php?topic=316446.0

Title: Re: Strange behavior: sending wrong signal with sensor data alone
Post by: Coding Badly on Apr 17, 2015, 03:53 am
The weird spikes are actually caused by missing bytes (characters here).
Why does the generated sine wave not have spikes?

Title: Re: Strange behavior: sending wrong signal with sensor data alone
Post by: yw5aj on Apr 17, 2015, 03:55 am
Why does the generated sine wave not have spikes?


The "wave" you saw is my finger pressing on it. The generated sine wave, as you can see in the code, is only having peak magnitude of 1, which is trivial compared to the sensor input.

Good question - sorry that I did not made it more clear in the OP. I will edit it now.
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 03:55 am
I just realized that this is indeed a different issue than the OP.
By your own admission, the topics are the same (dropped data).

Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 03:56 am

Threads merged.

Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 17, 2015, 04:52 am
I found an easier way to repeat the problem.

Code: [Select]

void setup() {
  Serial.begin(2000000);
}

void loop() {
  Serial.println(234);
}



Keep sampling with your favorite tool for 10,000 data points - there will be spikes.

Shawn
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 17, 2015, 05:11 am
Hi Coding Badly,

I feel that this post deviated far enough from the OP, as I have made progresses to narrow down the source of the problem. People will get lost if I keep posting here. Could you please help review whether the following post is OK to create a new one? It would be OK to say no too and I really appreciate your work in help organizing this forum.


Subject: Sending constant int from Arduino to PC causes error

I am using Arduino Uno Rev 3, and according to this link  (http://arduino.stackexchange.com/questions/296/how-high-of-a-baud-rate-can-i-go-without-errors)we can achieve 2000000 baud rate.

This problem can be easily repeated by the following code:

Code: [Select]

void setup() {
  Serial.begin(2000000);
}

void loop() {
  int val = 234;
  Serial.println(val);
}


I am using the serial-USB cable. To read the data, I used Python pyserial (< 20 lines of code including empty lines and import lines):

Code: [Select]

import serial
import matplotlib.pyplot as plt

LEN = 10000
data = []
ser = serial.Serial('COM3', baudrate=2000000)
for i in range(LEN):
    try:
        data.append(int(ser.readline()))
    except:
        pass
ser.close()

# %% Plot data
plt.figure(figsize=(6, 4))
plt.plot(data)
plt.savefig('data.png')


Here is what I got:
(http://i.stack.imgur.com/sFZUG.png)
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 07:05 am
Could you please help review whether the following post is OK to create a new one?
Really?  Rather than helping you with your problem you want me to waste my time justifying my moderation decisions?  I will pass.

Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 17, 2015, 07:12 am
Really?  Rather than helping you with your problem you want me to waste my time justifying my moderation decisions?  I will pass.


Don't take me wrong please! Again, as I said in the post above, I really appreciate your time and help in organizing the forum threads. Instead of trying to fight, I was thinking whether that would be a better way to discuss this issue. I will not post a new thread then.

Please don't misunderstand me, who truly appreciate all your effort!

Shawn
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 07:14 am

I cannot get it to fail (excess 234s trimmed)...

Code: [Select]

C:\ >C:\Python34\python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import serial
>>>
>>> LEN = 10000
>>> data = []
>>> ser = serial.Serial('COM3', baudrate=2000000)
>>> for i in range(LEN):
...     try:
...         data.append(int(ser.readline()))
...     except:
...         pass
...
>>> ser.close()
>>>
>>> data
[234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
....
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234,
 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234, 234]
>>>
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 07:23 am

...which brings us full circle to Reply #1 (http://forum.arduino.cc/index.php?topic=316217.msg2189353#msg2189353).  Your Python script is not removing data from the serial buffer fast enough.  You need some combination of: faster PC, better kernel driver, faster Python run-time, different computer-side programming language, framing, error detection.

Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 17, 2015, 07:36 am
...which brings us full circle to Reply #1 (http://forum.arduino.cc/index.php?topic=316217.msg2189353#msg2189353).  Your Python script is not removing data from the serial buffer fast enough.  You need some combination of: faster PC, better kernel driver, faster Python run-time, different computer-side programming language, framing, error detection.


Thanks a lot for testing it Coding! I have two thoughts though,

1) One reason that makes me think it is not a issue from slow computer/connection is my Reply #8 (http://forum.arduino.cc/index.php?topic=316217.msg2190602#msg2190602), where I added a very small sinusoidal curve to the int, and it worked perfectly. It is still a high baud rate, so if the speed is an issue, it would fail there too. If you don't mind, could you please take a look at that thread's code? I find that phenomenon really weird to understand (looks like the data type was automatically promoted by adding a value to it, but then again it is still int).

2) There is another slight chance (I might be totally wrong): because the error rate is relatively low, I only find out the "spikes" by plotting them (eye-balling them through the IDLE console does not work). If you do
Code: [Select]

for datum in data:
    if datum > 234:
        print(datum)

Would any value be printed out? (I might be wrong though)

Thanks again - I do appreciate your time into this very much!

Shawn
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 07:57 am
1) One reason that makes me think it is not a issue from slow computer/connection is my Reply #8 (http://forum.arduino.cc/index.php?topic=316217.msg2190602#msg2190602), where I added a very small sinusoidal curve to the int, and it worked perfectly. It is still a high baud rate, so if the speed is an issue, it would fail there too.
You increased the amount of work the Arduino must do between println calls.  That slows the data rate.  It slows the data rate enough that your computer can keep up.

The code is functionally equivalent to this where PauseForSlowComputer is high enough that the computer can keep up...

Code: [Select]
#define FASTADC 1  // Flag for prescale 16
// Code pasted for modifying ADCSRA
#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

void setup() {
  Serial.begin(1000000);
  #if FASTADC
    // set prescale to 16
    sbi(ADCSRA,ADPS2) ;
    cbi(ADCSRA,ADPS1) ;
    cbi(ADCSRA,ADPS0) ;
  #endif
}

const unsigned PauseForSlowComputer = 50;

void loop() {
  int val;
  val = analogRead(A0);
  delayMicroseconds( PauseForSlowComputer );
  Serial.println(val);
}
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 07:58 am
2) There is another slight chance (I might be totally wrong): because the error rate is relatively low, I only find out the "spikes" by plotting them (eye-balling them through the IDLE console does not work). If you do ... Would any value be printed out? (I might be wrong though)
This has been running for about 10 minutes (precise capture of the screen)...

Code: [Select]
C:\windows\system32 >C:\Python34\python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import serial
>>>
>>> count = 0
>>> ser = serial.Serial('COM3', baudrate=2000000)
>>> ser.flush()
>>> while True:
...   count = count + 1
...   line = ser.readline()
...   if line != b'234\r\n':
...     print( '#', count, ':', line )
...
Title: Re: Erraneous transmission at high baud rate for serial communications
Post by: yw5aj on Apr 17, 2015, 08:28 am
You increased the amount of work the Arduino must do between println calls.  That slows the data rate.  It slows the data rate enough that your computer can keep up.

The code is functionally equivalent to this where PauseForSlowComputer is high enough that the computer can keep up...

Code: [Select]
#define FASTADC 1  // Flag for prescale 16
// Code pasted for modifying ADCSRA
#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

void setup() {
  Serial.begin(1000000);
  #if FASTADC
    // set prescale to 16
    sbi(ADCSRA,ADPS2) ;
    cbi(ADCSRA,ADPS1) ;
    cbi(ADCSRA,ADPS0) ;
  #endif
}

const unsigned PauseForSlowComputer = 50;

void loop() {
  int val;
  val = analogRead(A0);
  delayMicroseconds( PauseForSlowComputer );
  Serial.println(val);
}


You are totally right! My goodness I should have thought about that. Thank you so much for your helpful experience! Finally got it: it worked when the sinusoid curve is there because it added a wait time between writing the serial port!

And - thank you for running the test too! Really appreciate it.

I will now update the title to show it has been answered.

Title: Re: [SOLVED] Erraneous transmission at high baud rate for serial communications
Post by: Coding Badly on Apr 17, 2015, 08:32 am

You are welcome.

Title: Re: [SOLVED] Erraneous transmission at high baud rate for serial communications
Post by: Robin2 on Apr 17, 2015, 11:47 am
Thank you Robin for your response! I did read your code before you post, but I thought that was a different kind of data being read. Mine was continuous data streaming from the sensor, with delimiters ("\r\n").
So use \r\n as a delimiter in place of the <> that my code uses. In fact you can probably just use \n on its own.

The second example in serial input basics (http://forum.arduino.cc/index.php?topic=288234.0) just uses an end-marker rather than start- and end-markers.

...R