Trying to read serial data sent by the board over serial (I'm at my wits end here)

I am attempting to write data from the board I am using (the Esperiff ESP8266) to my PC and read the data with pyserial. I've broken what I want to do down to just this aspect but it won't seem to read. And yes it does print properly on the serial monitor and yes pyserial is able to open the port with no errors. On the arduino IDE side my code is:

void setup() {
    Serial.begin(9600);
}

void loop() {
    //neither Serial.print nor Serial.write work either
    Serial.println("Why isn't this working???");
    delay(2000);
}

And for pyserial:

import serial
#yes i'm on windows
ser = serial.Serial('COM8', 9600, timeout=20)
#testing both and neither work
ser.read()
ser.readline()

I'm seriously stuck and don't know what I'm doing wrong. I have also tried restarting, using different cables and ports (changing the port name accordingly of course), but it still won't work. If any of you can help me figure this out, that would be greatly appreciated.

Have you tried first to receive on the IDE Serial monitor to confirm everything is working on the hardware / Arduino end ?

Yes, it prints to the serial monitor just fine.

Try setting the timeout to 1000, or bigger.

Note the serial output of a ESP device is only 3V3, and your PC might need a 5V signal on the serial port So you would need to add a level shifter on the output of the ESP.

Don't try to look at the serial port with the serial monitor otherwise it will not send any signal out to your PC.

1 Like

This is important

You can do the Python side without any kind of delay if you use a while loop

import serial

ser = serial.Serial()
ser.port = 'COM8'
ser.baudrate = 9600
ser.open()

while True:
    if ser.isOpen():
        if ser.inWaiting():
            try:
                #ASCII string
                inp = ser.readline().decode('utf-8')
                #bytearray
                #inp = ser.readline()
                print(inp)                       
            except Exception as e:
                pass

I close Arduino IDE when I run pyserial otherwise. I can't open the port on pyserial when Arduino IDE is open, it throws and error.

Not helping

It should be sufficient to close serial monitor and/or serial plotter.

Only one application can have the serial port open at a time. Applications involved in this case:

  1. The upload tool.
  2. Serial monitor / serial plotter.
  3. Your python application.

The IDE itself is not an application that uses the serial port, only (1) and (2) above that are controlled by the IDE.

@tstein19 ok that's good we have established that the serial monitor is closed when you try to run your Python code but what about the piece of code I posted, does it run , does it produce errors or does it do nothing. The try: / except: is there to hopefully ignore any errors you might get and print your string from the 8266

It just outputs an empty byte array i.e.

b''

I should also probably mention that I'm not running this a script, but from the terminal so I do it line by line.

@tstein19 I do it line by line also and it works

Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import serial
>>> ser = serial.Serial()
>>> ser.baudrate = 9600
>>> ser.port = "COM8"
>>> ser.open()
>>> ser.rts=True
>>> while True:
...     if ser.inWaiting():
...             try:
...                     inp=ser.readline()
...                     print(inp)
...             except Exception as e:
...                     pass

if you use the same usb cable that shows data in the serial monitor the above should also show data in the terminal, I don't understand why your app is printing 'inp' as a bytearray did you opt for a bytearray over ascii as a display. Try again and make sure the indentation is correct or perhaps even better try it with the Python idle IDE.

BTW I did add ser.rts = True to reset the Arduino on making a connection

@tstein19 I can't see anywhere in this thread where you have said how you have wired the serial connection from the ESP8266 to your PC. Can you show us a diagram or a photo? Details of any adaptor you are using at the PC end e.g. USB-TTL serial or USB-RS232 would be helpful. Or maybe your PC has a 9 way serial port connector.

I tried this and it’s just stuck in an infinite loop that outputs nothing. As an aside I changed the if statement to

if ser.in_waiting:

since the pyserial docs have that listed as a property now instead of a function.

try this simple terminal emulator

# terminal.py - simple terminal emulator - requires pyserial

import serial
import sys
import msvcrt
import time

serialPort = serial.Serial(
    port="COM16", baudrate=115200, bytesize=8, timeout=1, stopbits=serial.STOPBITS_ONE
)
serialPort.rtscts = False
serialPort.xonxoff = False
serialPort.dsrdtr = False
time.sleep(3)
sys.stdout.write("Python terminal emulator \n")
serialPort.write('hello terminal\n'.encode())
while 1:
    try:
        # serial data received?  if so read byte and print it
        if serialPort.in_waiting > 0:
            char = serialPort.read()
            sys.stdout.write(str(char, 'ASCII'))
        # keyboard hit? if so read key and transmit over serial
        if msvcrt.kbhit():
            char = msvcrt.getch()
            serialPort.write(char)
    except:
        pass

Arduino test program (i used an ESP32 but should work on many microcontrollers)

// terminal emulator test
// read characters from serial echo printable

void setup() {
  Serial.begin(115200);
}

void loop() {
  if (Serial.available()) {
    char ch = Serial.read();
    if (isprint(ch)) {
      Serial.print("received ");
      Serial.println(ch);
    }
  }
}

open Windows Command Prompt (make sure Arduino IDE Serial Monitor is closed)
run python with above code - characters entered are echoed

F:\Programming\python\Serial>python3 terminal.py
Python terminal emulator
received h
received e
received l
received l
received o
received
received t
received e
received r
received m
received i
received n
received a
received l
received t
received e
received s
received t
received 1

That didn't work either, it's just stuck on

Python terminal emulator

What do you actually mean by stuck?
Is it an error message during compile? If so is that the exact message?

Ok so get this, I went into the device manager to see if adjusting the settings would do anything. And under the advanced settings there is an option called "Disable ModemHandShake". I tick this option and lo and behold, IT WORKS NOW!!! I feel stupid for not trying this sooner, and thanks to everyone for their help. I hope my misfortune can help others with the same problem.

ran your post 1 program on a ESP8266 modified post 14 port and baudrate and it works with or without "Disable ModemHandShake" ticked - very strange?