Unusual characters returned to Python via serial

Greetings.

I built a Rubik's Cube robot, had to incorporate a Raspberry Pi for the brain work. The Pi sends a command code to the Mega, Mega sends a character back to Pi when the stepper motors are done, Pi thinks about it, sends the next command code, etc, etc, ad nauseum. It worked great, but now, suddenly, the serial communication is weird.

I swapped out Mega boards, but get the same result. I switched to my Linux laptop in case the Pi board is whacked. Same result. (I hope the Pi isn't whacked, and screwed up both of my Mega boards.) I also had a terrifically bad time uploading to both Mega's, and finally got success by going into the Sketch menu and hitting Upload from there, on the laptop at least. Weird...

I wrote a simple test program in Python and Arduino. The Mega should send back the letters T-E-S-T-I-N-G, one at a time, and keep doing so because it is in the void() loop. Mega sends back 1-2-5-9-2, then the letters, from a random place in the loop, but then it continues as normal. I tried different baud rates, but no difference. In the Rubik's program, the Mega sends those exact 5 numbers, but then just keeps sending numbers, and odd letters.

Here is the Python code:

import serial
import time
import sys

robot = serial.Serial('/dev/ttyACM0',9600)
time.sleep(5)

def main():
    robot.write(b'17D')
    time.sleep(1)
    
    time.sleep(1)
    tileColor = 0     
    while(tileColor == 0):
        tileColor = robot.read()
        time.sleep(1)
        print(tileColor)
        tileColor = 0
        robot.write(b'17D')
        time.sleep(1)
    
    return


    ##############  START OF PROGRAM!!!  ##########################

main()

Here is the Arduino code:

char command;

void setup() {

  Serial.begin(9600);
  while (!Serial){
    ;
  }
  delay(1000);
  Serial.print('10');
  delay(1000);
}

void loop() {
  command = 0;
  
  if (Serial.available()) {
    command = Serial.read();
    delay(1000);
    Serial.print('t');
    delay(1000);
  }

  if (Serial.available()) {
    command = Serial.read();
    delay(1000);
    Serial.print('e');
    delay(1000);
  }
and so on until t-e-s-t-i-n-g are all sent, then it loops

And here is what the Mega sends back:

====== RESTART: /media/mark/USB STICK/Correct Files To Run/test_serial.py ======
b'1'
b'2'
b'5'
b'9'
b'2'
b'n'
b'g'
b't'

I stopped it here, but it continues sending the proper letters, no more numbers.

I started building this robot over 10+ years ago, one of those evolving projects (still is), and it worked great, until now.

I have done a lot of researching about this, especially why those exact 5 numbers every time, with no success.

And now I am here! :oD

What? Something strange to me here...

If with that '10' you meant to send the string "10" (two characters) you should use double quotes (the default delimiters for a string), not that single one (it's used for single characters), thus:
Serial.print("10");
If you use that wrong syntax, you can't get what you need, and I think that "12592" is printed out by that first Serial.print() (who knows what the compiler did to interpret that '10' as a string...).
In fact if you cut the sketch out to this:

char command;

void setup() {

  Serial.begin(9600);
  while (!Serial){
    ;
  }
  delay(1000);
  Serial.print('10');
  delay(1000);
}

void loop() {

}

you'll get that "12592" string as the only output.

Next, even if possible, if you need to send single characters/bytes you should use Serial.write() and not Serial.print():

  if (Serial.available()) {
    command = Serial.read();
    delay(1000);
    Serial.write('t');
    delay(1000);
  }

You can still use Serial.print() to print single characters, but it isn't a good thing to do, you better explicitly use "write".

That said, I think your test code could be changed and shortened this way:

char command;
char resp[] = "testing";
byte r = 0;

void setup() {

  Serial.begin(9600);
  delay(1000);
  Serial.print("10");
  delay(1000);
}

void loop() {
  command = 0;
  
  if (Serial.available()) {
    command = Serial.read();
    delay(1000);
    Serial.write(resp[r++]);
    if (r > strlen(resp)-1) r = 0;
    delay(1000);   
  }
}

Does this work to you?

Thank you, @docdoc, for working with me on this. I put in your code, and it yielded this result: b'1', b'0', b't', b'e', etc, then repeating the "testing" characters continuously. (I found another Mega. I think the Pi fried the boot loader on the other two.)

This will be long, but hopefully it will be of value to someone else trying this sort of thing.

I would like to explain my program pair functioning a bit more, and it will make more sense. Being a self-taught programmer (of sorts), I research out info on what I want to accomplish, collage code bits together to get some result, then sleek out the collage into a sensible arrangement, and in my robot's case, I keep adding to it and refining it. That is where the '10' came from, and it stuck.

So, I have the Arduino code uploaded to the Mega, then I run the Python program. The Mega is waiting in void setup() for a signal from the Pi, not caring what it is, only that the serial connection is working. Then Mega sends something back to Pi, again not caring what it is (this being the '10', until now). It is just confirming to Pi that it has accomplished the command it received, and is now waiting for the next one. It does this with every command received.

The Pi doesn't care what it gets, so long as it is not a 0 of any sort, because that is what it sets its receiving variable to, waiting for a change.

Arduino code notes:
Serial.print("10") results in a character mismatch error, with the double quotes.
Serial.print('10') results in Pi printing the 1-2-5-9-2, then the t-e-s-t-i-n-g repeating.
Serial.print(10) results in b'1', b'0', then the t-e-s-t-i-n-g repeating.
Serial.print(1) also results in b'1', b'0', then the t-e-s-t-i-n-g repeating.
Serial.print('1') results in just b'1', then the t-e-s-t-i-n-g repeating.
Serial.write(1) results in b'\x01', then etc.
It must be a number, because a letter gets the character mismatch (or somesuch) error.
A delay(1000) at the end of void setup() AND at the beginning of void loop() is needed for the t-e-s-t-i-n-g to start correctly, and a minimum for at the end of each letter sent is delay(750), or letters get skipped. I will set them all to delay(1000) in my sketch. The Serial transfer seems to take a long time.

Right now, every command sent has a '10' response, so they will all be changed to '1' to save processing time.

On the Python side:
The robot.write(b'17D') MUST be a number, presented exactly in this syntax, or various errors occur. ("robot" is the variable for the Serial connection.)

A curious thing happens if that line is commented out: the compiler highlights the number and throws the error "invalid decimal literal". That is in the Idle gui, and I am curious if some other gui like Thonny, which is the Raspberry Pi gui, will do that as well. I will certainly try that out down the line!

So, now I get to learn how to burn the bootloader from one Arduino to another.

I have learned so much from this forum that I am very pleased to be able to contribute some knowledge back into the community.

@docdoc, I also want to thank you for working up that code bit. While I intentionally sent one character at a time for testing purposes, I will be experimenting with your code for future use.

The argument to Serial.print('10') is treated as a 16 bit integer composed of the two ASCII characters for '1' and '0' (0x31 and 0x30) or 0x3130 in hex representation, which is equivalent to 12592 in decimal representation.

@dasrod Serial.print() is intended for formatted, human readable output. Use Serial.write() for binary, unformatted output.

@jremington, thanks for the 12592 numbers explanation.

All the command and response numbers (or characters) going back and forth between the Python and Arduino are never seen. I only print them out for troubleshooting, and I know what they should be.

It really doesn't matter if they are integers or characters. It works either way.

I appreciate the Serial.print and Serial.write explanations. That knowledge will be good if I undertake a project that requires text transfer.

Now to burn the bootloaders and hope that will fix those 2 Mega boards.

Cheers!

It really doesn't matter if they are integers or characters. It works either way

Among other things, I was trying to explain why you seemed to be having trouble understanding results like the following.

Was that simply misleading information?

No, not misleading information. It turns out the Serial.print(b'10D') was not the issue at all, but rather damaged hardware was. The '10' was only a symptom, and a great learning experience.

So, I wanted to try all the various combinations, and I listed out the results. This was a purely empirical experiment to satisfy my curiosity, and I discovered that sending a '1' instead of a '10' will save a lot of time over the 150, give or take, moves required to solve the cube. Maybe I am wasting forum space with this, but it may be quite useful to someone else like me - a dabbler, basically.

I currently have no need for intimately knowing the difference in .print and .write, and this old brain can only hold so much. But now, if/when I do need to deep dive into it, I know that .write is for text, so I will have that starting point. And I thank you for that.

In the Python program, the "D" in (b'17D') has to be there. I believe it is the end of transmission signal, but I can not find any reference to that anywhere. But I do know that things won't work without it, and I am definitely not going to try to fix things that aren't broken!

I said that I am a "dabbler programmer", but that isn't quite accurate. I don't have the depth of a professional programmer, but between these two programs I have over 8,000 lines of code.

And it works! My robot solves the Rubik's Cube, and I am very proud of that.

I could not have done it without the help of folks like you, who share their knowledge. Again, thank you!

If you mean you need to send a byte with decimal value 10 (or 0x0A in hex) you must use "Serial.write(10);" without any quote.
A byte less will speed it up? I'm not convinced about that, at 9600 baud it's 1/960th of a second saved. Not much.
Anyway, if the serial communication is a key factor to you, increase the speed at 119200 baud (on both sides...).

Well, I don't know your age, but I'm 65 and I don't feel this old. And yes, I'm sorry to say if you want to learn and understand Arduino programming you should know the difference between .print and .write, together with number representations, strings, etcetera.

That's not good: if you need to let (digital, aka codes) parties to communicate you must know the protocol and its purpose/meaning. I know nothing about Python, but if you say you need to send a "D" it means a string made up by a byte representing the letter "D" (0x64 hex, or 100 decimal). I don't know what you mean with "b'17D'", it doesn't look like either a "D" or a byte (17D hex is 2 bytes, 01 and 7D). You said it's a "end of transmission", so it could mean sending 0x0D: it's not a "D" but a byte with value 0x0D or 13 decimal, used to end a line. known as CR (Carriage Return), or the char '\r'.
what I'm saying is you must first know what you need to do and how the elements you're using work.

I'm happy to see you've got your goal to let that robot work, but I still recommend you to study a bit more. :wink:
Cheers!

I watch the data coming back from the Arduino on the Python monitor screen. It takes about 1 second for each of the 1-2-5-9-2 characters to comes back. So that means about 4 seconds extra for each of the perhaps 150 moves to solve the cube. That is 10 minutes or so. It takes about 15 to 20 minutes or more to solve the cube. If I can cut that time in half, that would be awesome.

Ten years ago when I worked all this out, I also monitored the data the Mega was receiving. (b'17D') would come in as '17'. Right now I can't remember if the D was end of transmission or end of line, but at that time it puzzled me that the D was not in some other control character format, but it worked, so I ran with it. I research on a need to know basis, because programming happens few and far between.

I only program now and then. I suppose I would be a beginner/intermediate in C++ and Python. I am not so great at sludging through all the nitty-gritty details of all the protocols. This is the only time I have used serial communication. Ever.

When you say it's not good that I don't know all the details of the protocol, I agree to some extent. I dig through it, but I just don't catch on so good. That's why I didn't pursue programming as a career: I recognized long ago my mental limits in this. Besides, I would rather make stuff than sit at a computer all day, hence, I am a retired Tool & Die Maker!

I suspect that most of the people who come to this forum for help are like me. I am a living room guitarist, and pretty good. Good enough to get on a stage and perform? Not even close. Same with programming.

This is the first time I have literally asked a question. I had to create an account to do it even! It is intimidating to do so, because I feel like a dummy.
The creator of the For Dummies books was a genius. He understood us. I learned a lot of programming from those books!

I have waxed eloquent about being an intimidated dummy for the sake of other intimidated dummies out there.

Take heart! Don't be afraid to ask! But do your homework first.

Don't be scared off by comments like "you should know". Sit up and listen! This is not meant as an insult! Folks like @docdoc and @jremington are here to help us! We don't know! They do, so listen to them and evaluate, just like I did.

@docdoc and @jremington, and all you other knowledgeable people who help us, thank you, thank you, and thank you again!

OK, enough of this.

In Python, b'17D' indicates a sequence of three bytes representing the ASCII characters '1', '7', and 'D'.

So, D is actually the ASCII value of the character, not the binary value 0x0D.

So, 'D' in the last place of the transmission must be interpreted by the Arduino serial library as a carriage return. It didn't occur to me until now to look at the Arduino documentation about it. That I will do when I can get to it. I read your earlier post about that again, and now it makes more sense.

The character 'D' has code 68.
The carriage return (CR) has code 13.

To transmit the characters '1', '7', and CR, you should write:

b'17\x0d'
or
b'17\r'

In this way, three bytes are transmitted, with values 49, 55, and 13.

But I'm not sure if that's what you want or need. It all depends on what the receiving program expects to receive.

I’ve written a small tutorial on interfacing with Python. See Two ways communication between Python3 and Arduino

Due to delay(1000). No doubt.
Changing to delay(500) should give you your 2x speedup.

Speaking about that "12592" string, it comes out as the output of the wrong, single "Serial.print('10')" instruction, there's no delay. So if you see those bytes with 1 second delay each, it looks like something wrong somewhere else, like the Python side or the serial connection.

Apart from this first string, if in your protocol you have one byte/char sent for each move the timings should be driven by the physical actuator movements, not by things line delay() in the code. Not knowing the "real" code for both sides (Arduino and Python) I can say nothing more than this, but if the speed is a requirement for you, my advise is to check the code(s) and avoid useless and/or length delays.

I have had more interesting developments here. Yes, the Raspberry Pi scrambled the bootloaders on two Mega's, which I have repaired. But running Python from my laptop gave different results than from the Pi. I thought maybe to update the Arduino IDE, which I haven't done until now, and see what happenes. Altogether different returns from the Mega after that.

It then occurred to me that over the 10+ years that I was actively working on this, I was using Python2. I now use Python3, and have upgraded the Arduino IDE a couple times. I have not been able to work with the robot for 3 or 4 years now.

So what used to work doesn't anymore. So, I am going to take the advice from you all and dig into the protocols of the serial communications. I will start with the tutorial that @J-M-L wrote, and go from there.

Just like haven't worked on the robot for 3 or 4 years because of life situations, so also it will be at least 2 weeks before I can get back to working on it now.

Thanks, everyone, and I'll post what happens later.

Tthings change, fact of life !

Have fun and taking it slow is fine .

Hi, everyone!

I am finally back with an update to my Python/Arduino serial communication problem in my Rubik’s Cube robot.

With the guidance from you fine people I have learned what I need to make it work.

Here is the Python snippet, with “robot” being my serial entity:

    robot.write(b'g\n')
    time.sleep(1)   

    arduinoReply = ""     
    while(arduinoReply == ""):
        arduinoReply = robot.read().decode('utf-8').rstrip()
        time.sleep(1)
        print(arduinoReply)
    arduinoReply = ""

And here is the Arduino snippet, which is in void loop():

  if (Serial.available() > 0){
    Serial.write("s");
    command = Serial.readStringUntil('\n');
    command_codes();
    delay(1000);
    command = "0";
  }

The Arduino waits for a string from Python, which is "g" in this snippet, then goes to the command_codes() function, finds that code in the list, executes that function, then goes back to "void loop()", sends an "s" to tell Python it has performed that task and is ready for the the next command.

When Python receives the arduinoReply, in this case an "s", but in this part of the code it doesn't matter what it is, only that it is a response, it then calculates the next move, sends the appropriate command code, waits, etc.

Thank you again, everyone!

Hmmm…. I need to learn how to properly use the code insert function.