Python Serial Read Lagging Behind Arduino Data Stream – How to Improve Real-Time Performance?

Hello there. I am trying to do a project in which I record analog pressure data from the Arduino Mega in real time and save it to excel using python with the openpyxl library. The Arduino sends data at a fast rate in the serial monitor but compared to the output of the python script, it is lagging behind by a large margin (By lagging, I mean that even when the water source is off, it is still logging pressure from when the water source was on). What can I possibly do in the python script or Arduino program to get a fast sampling rate and record it in real time without lag?

Arduino code:

const float  OffSet = 0.475 ; // adjust this value
int analog = A0;
 
float avgV = 0.0;
 
float V=0;
float P;
 
void setup()
{
  Serial.begin(115200);        // open serial port, set the baud rate to 9600 bps
  pinMode(analog, INPUT);
}
void loop()
{
  //Connect sensor to Analog 0
 
  V = analogRead(analog) * 5.00 / 1024;     //Sensor output voltage
  
 
  P = (V - OffSet) * 250;             //Calculate water pressure
 
  // Serial.print("Voltage:");
  // Serial.print(V, 3);
  // Serial.println("V");
 
  //Serial.print(" Pressure:");
  Serial.println(P, 1);
  //Serial.print(" KPa");
  //Serial.println();
  delay(10); //debugging purposes
 
}

Python code:

import serial
import openpyxl
from datetime import datetime
import os

ser = serial.Serial(port='COM7', baudrate=115200, timeout=None)

file_name = "Datasetpressure.xlsx"

# Check if file exists, if not create a new one with headers
if not os.path.exists(file_name):
    wb = openpyxl.Workbook()
    ws = wb.active
    ws.title = "Pressure Data"
    ws.append(["Timestamp", "Pressure (kPA)"])  # Add header row
    wb.save(file_name)
else:
    wb = openpyxl.load_workbook(file_name)
    ws = wb.active

try:
    print("Recording data... Press Ctrl+C to stop.")
    while True:
        if ser.in_waiting > 0:
            line = ser.readline().decode('utf-8').strip()
            try:
                flow_rate = float(line)
                timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

                ws.append([timestamp, flow_rate])
                wb.save(file_name)

                print(f"{timestamp} - Pressure: {flow_rate} kPa")
            except ValueError:
                print(f"Invalid data received: {line}")

except KeyboardInterrupt:
    print("\nStopping data recording...")

finally:
    ser.close()
    wb.save(file_name)
    print(f"Data saved to {file_name}.")

Don't save after every reading?

@jamesalgonquin you could save to a .csv file to be opened after your logging session. If data is coming in that fast your not going to read and make sense of the values on a spreadsheet anyway.

Part of that delay may be because Python is not compiled but is interpreted.

Python is about an order of magnitude slower than C/C++, and often unsuitable for real time data acquisition. Either buy a much faster processor or stay with the Arduino IDE and C/C++.

1 Like

As @jremington recommends a faster processor would help. Look at one of the ESP units.

Thanks for the inputs! I'll try logging the data into a csv file instead of an xls file and try to use an ESP32.

Hi, @jamesalgonquin
Welcome to the forum.

Have tried "Data Stream" directly to Excel ?
We have a programmer at work, he does data logging and storage and analysis all in Excel, no Python.

Sorry I'm not an Excel or data stream expert so you may have to research how to make it do what you want.

Tom.... :smiley: :+1: :coffee: :australia:

I’ve written a small tutorial on interfacing with Python. See Two ways communication between Python3 and Arduino

Maybe that can help


To your particular issue, avoid saving to Excel in real-time because libraries like openpyxl are too slow.

Instead, buffer incoming data in memory and save it in batches, maybe every 1000 samples and a faster format like CSV instead of Excel to reduce overhead.

If performance is critical, consider logging raw data to a binary file using NumPy or another efficient format and converting it later. Also, ensure that the serial reading is not blocked by any UI or print operations, which can also slow things down.

Alright so I just finished doing the trials. @J-M-L a and @sumguy 's advice worked! Saving it to a csv file instead of an xlsx file made the data acquisition run in real-time in the output of the python IDE and save the data without problems :grinning_face: :+1:

1 Like

Haven't tried this since for some reasons, my excel doesn't have it and installing that feature causes an error... Maybe I'll check it out when I get the chance. :3

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.