Non ASCII bytes are misinterpreted

Projects Interfacing w/ Software on the Computer
1

Hi there

I'm using a simple Python code to send serial data to Arduino Mega in order to move a servo motor.
Because a servo motor can yaw from 0-180 degrees, I would like to use only one byte to carry the angle.
I've found that this can be done in Python using chr(angle).

This is my Python code :

# -*- coding: UTF8 -*-
import serial
import time
ser = serial.Serial("COM25", 9600)
print ser.isOpen()
time.sleep(2)
msg = ('#$%' + str(chr(200)) + str(chr(10)) #Servo angle
+ 'C' + str(chr(255)) + 'C' + '1' + '0' + '0' + '1' + str(chr(182)) + '%$#')
ser.write(msg)
print ser.readline().rstrip()
for i in range(14):
        print ser.readline().rstrip()
ser.close()

And the part that is related to Serial communication in my Arduino code is :

void loop()
{
// Loop Code.
	// Check if the message has completely been received.
  	if (msgComplete) {
  		// Toggle the flag to false.
  		msgComplete = false;
  		// Extract the message and remove the delimeter.
        msg = msg.substring(msg.indexOf("#$%")+3,msg.indexOf("%$#"));
        // FOR DEBUGGING: respond to message arrival.
        Serial.println("Message arrived: " + msg);
        control(msg);
        msg = "";
	//Serial.flush();
    }
}

void serialEvent()
{
  	char inChar = Serial.read();
	msg += inChar;
	if (msg.indexOf("%$#") > 0)
        {
		msgComplete = true;
        }
}

The control function decodes the message (meaning that is extracts useful information from each part) .

dir = (int)(msg.charAt(1));
Serial.println("direction = " + String((int)dir));

And writing to Servo in another function :

void move(byte sp1, byte dir, char status)
{
	DIR_SERVO.write(dir);
	Serial.println(dir);
        delay(150);

The output for the previous Python program is (I didn't write the whole program here so some lines of code print outputs that I didn't mention) :

True
Serial Communication initialized ...
[Decode error - output not utf-8]

[Decode error - output not utf-8]

sp1 = 200
direction = 10
thrusters1 = C
sp2 = 255
thrusters2 = C
Arm1 = 1
Arm2 = 0
Laser = 0
Light = 1
Camera = 182
200
10
[Finished in 2.6s]

Ok, the problem is, when I send a value like 170 or 180 (non ASCII characters, remember that I turn them into characters to be able to send them in one byte), this message appears (in sublime text in the console):

[Decode error - output not utf-8]

Ok, I have no choice, I have to send angles from 0 to 180, and values above 128 are not allowed in ASCII.

I forgot to mention that, in Arduino Serial monitor, the ASCII characters are always sent incorrectly. Like when I write ALT+ 5 or ALT+9, they are sent as 63 in both situations !! although different symbols appear !

So guys, I'm really confused with this problem, I want to send the angle in only one byte regardless of sending them in ASCII or anything else, I don't care. But if the only way to send them is to convert them into a char symbol, and this is misunderstood by Arduino, I need to know how to send non ASCII characters so that Arduino can properly understand them.

Note that, when I send non ASCII characters to arduino using my Python code, although the decoding error message appear in console, the number is properly sent; however, the servo don't move in the way it supposes to do. It move to the center first then goes to the desired angle, and sometimes don't !!!

So, any help please ?

2

I haven't gone through your code - there is a lot of it.

What is producing the error [Decode error - output not utf-8]

It doesn't seem like an Arduino error.

There should be no problem sending any value of byte to the Arduino.

Is it possible the error arises because you are trying to put 180 into a char (rather than a byte) which may only accept values up to 127.

...R

3

On the PC, a byte with the high order bit set (any code greater than 127) can be interpreted as the beginning of a UTF-8 encoded sequence. These sequences allow you to send, for example, Chinese or Japanese characters and have them interpreted correctly.
I'm not familiar with Python but there's probably a way for you to tell it that you are reading raw bytes rather than encoded byte sequences.

Pete

4

I'm concatenating each incoming character to the msg String object. So when arduino receives a non-ASCII symbol like chr(0) that yields nothing at all - it converts it into ASCII to be able to add it to a String.
I think this is the problem. I do really send Arduino normal bytes, but the problem is that it changes them into ASCII characters.

5

What you're doing on the Arduino end is irrelevant. The error is from the Python script which is complaining that the string you are trying to send to the arduino is not a properly formatted UTF-8 string.

In this line of code:

msg = ('#$%' + str(chr(200)) + str(chr(10)) #Servo angle
+ 'C' + str(chr(255)) + 'C' + '1' + '0' + '0' + '1' + str(chr(182)) + '%$#')

the chr(200) and chr(182) are inserting non-ascii characters into the string which make python think it is a UTF-8 string.
At the beginning of the Python script it says:

# -*- coding: UTF8 -*-

I don't know if this is a comment or a command of some sort but you need to find a way of telling python NOT to use UTF-8 encoding. You basically want a plain binary encoding, sometimes referred to as RAW.

Pete