Reading and writing raw data to Arduino

Projects Programming
1

Hi!

I have an Arduino program that listens for data from serial. Ideally this data comes in as raw binary, not text from the serial monitor, as I don't really want to have to process a lot of text on the Arduino. For this I have written a Python program using pyserial to handle sending data. The problem is that when the Python program is connected to the Arduino's port, no other serial monitor can log whatever data the Arduino sends back. I've tried hard to make the Python program print out serial output coming from the Arduino, but it never works well timing wise considering it's constantly waiting for an input (even with threading).

Has anyone dealt with a similar situation? Are there any solutions?

Thanks in advance!

2

What Arduino are you using?

Your best solution is to use a board with an additional hardware serial port which can connect to a terminal program on a PC using a TTL to USB converter module. They are inexpensive and are an important addition to your Arduino parts pile.

If you have a board like a UNO without this additional port, you can use software serial to connect the rx and tx pins to the converter.

3

That is a restriction of the PC operating system. it is possible to implement serial port sharing, but you will have to download additional PC software to enable that option.

For troubles with your Python program, have a look at this contribution from a forum member: Two ways communication between Python3 and Arduino

4

have a look at this simple Python terminal emulator for Windows
it shows how to receive data from an arduino and display it on a Windows Command Prompt

# terminal.py - simple terminal emulator - requires pyserial

import serial
import sys
import msvcrt
import time

serialPort = serial.Serial(
    port="COM24", baudrate=115200, bytesize=8, timeout=1, stopbits=serial.STOPBITS_ONE
)
serialPort.rtscts = False
serialPort.xonxoff = False
serialPort.dsrdtr = False 
sys.stdout.write("Python terminal emulator \n")
serialPort.write('hello terminal\n'.encode())
while 1:
    try:
        # serial data received?  if so read byte and print it
        if serialPort.in_waiting > 0:
            char = serialPort.read()
            #sys.stdout.write("\nreceived ")
            sys.stdout.write(str(char, 'ASCII'))
            #serialPort.write(str(char, 'ASCII'))
        # keyboard hit? if so read key and transmit over serial
        if msvcrt.kbhit():
            char = msvcrt.getch()
            if char == u'\x1b':
                sys.stdout.write("Exiting")
                quit()
                exit(0)
            serialPort.write(char)
            sys.stdout.write(char.decode())
    except:
        pass

the section which reads keyboard input you would replace with the code to transmit your binary data

5

Once your other issue is resolved, how do you plan on handling data synchronization when receiving raw binary serial data? Simple Start / End markers won't work as they could contained in the data. Typically, a more sophisticated scheme is required with packetization, framing, checksum / CRC, etc.

6

Thanks everyone for the quick answers! I'm still working on my first problem, as I have an UNO and I'm trying to fit horace's implementation to my needs.

As to handling data sync, honestly I haven't thought about that. I must say that I'm a bit of a beginner so my plan so far was to catch anything that isn't just zeros as the start of the message, and look for an end marker (or actually just limit the message length, so just by looking at the start marker I know only 4 bytes are relevant, so just use them)

7

Thanks so much for this snippet, this really saved me. It works now and so far I'm not really having any trouble with finding the start and end of data, as it comes in a standard format I've chosen.

8

That's not at all robust. There are many ways for it to go off the rails.

What happens when TX and RX are out of synch and the RX picks what it thinks is the "first" byte but it's actually in the middle of the packet?

Also, remember, bytes in raw binary data can be any value.
What happens if a Start / End marker appears in the middle of the data packet? What happens if the first byte of the data packet is actually zeros?

9

Well, I guess you're right :sweat_smile:
To be honest this whole thing was a bit of a tangent in ny project just for some debugging (which looking back took a bit too much time lol).

But if I ever wanted this to be an actually reliable communications method I'd have to figure something out. For now I just know that the first byte is one of a number of lowercase characters, so it's non-zero, and following that starter come just a few numbers (a knowm quantity), it's not that I'm sending say images or audio or something, so as long as I'm aware of the exact format it might be just reliable enough for testing :slight_smile: but yeah, if I were sending data that was in any measure more complex this probably wouldnt do it.

Thanks for the help!

10

One thing that would help is if the gap between successive packets was much larger than the gap between any two bytes within the packets. That way, you could key off the gap to identify the start of packet. Also, you should either know the packet size in advance or send it in the beginning of each packet. That way, you'll know there was a transmission error if you encounter the inter-packet gap before receiving the entire payload. Also, you should have a checksum or CRC at the end of the packet. If it fails, discard the whole packet as errored. If you receive too many bytes before the large gap, again discard the entire thing as failed.

11

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.