Need some help with serial timing for auto calibration with test equipment

I don't know if this belongs here, or another place.

But I am banging my head trying to retrieve from my serial output in time with a python script that is controlling my test equipment. This setup is doing an auto calibration for me. I have two different python scripts, one that calibrates voltage and the other that calibrates current. I got the voltage calibration to work and output into a speadsheet. But the current calibration I can't get the timing right. I've tried continuously outputting serial data and timing it like I am now. I have tried quite a few different values in my python script for sleep as well.

Anyway. Here is the relevant section of the serial output from my sketch:

void SerialData() {
 #ifdef CAL_DISCHARGE
  serialTime = millis();
  if (serialTime >= serialTime0 + 2500) {
    serialTime0 = millis();

    // Calibrate for Cell 1 voltage
    if (C1.cell > 0.90) {
       Serial.println(C1.cell, 6);
      // Calibrate for Cell 2 voltage
    } else if (C2.cell > 0.90) {
      Serial.println(C2.cell, 6);
      // Calibrate for current
    } else {
      Serial.println(Icurrent);
      if (Serial.available()) {        // Check if data is available to read
        char command = Serial.read();  // Read the incoming byte as a character
        if (command == 'C') {          // Check for a specific command
          calibrateQOV();              // Call a function when the command 'C' is received
        }
      }
    }
  }
#endif  // end CAL_DISCHARGE

Here is my python script that works for voltage:

import pyvisa
import time
import serial
import pandas as pd
import numpy as np
from pymeasure.instruments.keithley import KeithleyDMM6500

# Define the IP address for the Keithley DMM
ip_address_dmm = "XXX.XXX.XXX.XXX"

# Initialize the DMM
dmm = KeithleyDMM6500(f"TCPIP::{ip_address_dmm}::inst0::INSTR")

# Initialize Serial

ser = serial.Serial('COM7', 115200)



# Initialize data storage
data_log = []
cell_voltages = []
dmm_readings = []

def main():
    try:
        time.sleep(1)
        # Initialize the VISA resource manager
        rm = pyvisa.ResourceManager()
        instadd = "TCPIP::XXX.XXX.XXX.XXX::INSTR"
        inst = rm.open_resource(instadd)
        inst.write_termination = '\n'
        inst.read_termination = '\n'

        # Turn on the output
        inst.write('OUTP CH1,ON')
        inst.write(f'SOUR:VOLT:SET CH1,(2.0)')
        time.sleep(1)
        
        dmm.auto_range()  # Adjusts range automatically

        # Sweep through voltage settings and measure with DMM
        for i in range(20, 46, 1):
            voltage = i / 10.00 # Convert to float
            inst.write(f'SOUR:VOLT:SET CH1,{voltage}')
            time.sleep(0.10)
            dmm_reading = dmm.voltage  # Read voltage from DMM
            time.sleep(0.1)
            data = ser.readline().decode('utf-8').strip()  # Read serial data
            
            # Convert serial data to float if possible
            try:
                cell_voltage = float(data)
            except ValueError:
                cell_voltage = None  # Skip if invalid

            if cell_voltage is not None:
                cell_voltages.append(cell_voltage)
                dmm_readings.append(dmm_reading)
            else:
                print(f"Invalid data received: {data}")

            print(f"Voltage set to {voltage}V, DMM reading: {dmm_reading}V, Cell Voltage: {cell_voltage}")

            # Append data to log
            data_log.append({
                "Set Voltage (V)": voltage,
                "DMM Reading (V)": dmm_reading,
                "Cell Voltage": cell_voltage
            })
            
            time.sleep(1)

        # Turn off the output
        inst.write('OUTP CH1,OFF')
        print("Measurements are done, closing connections")

    except Exception as e:
        print(f"An error occurred: {e}")

    finally:
        # Ensure the instrument connection is closed
        try:
            inst.close()
            dmm.close()
        except NameError:
            pass  # If inst was never initialized due to an earlier error

        # Compute the best-fit polynomial
        if cell_voltages and dmm_readings:
            poly_degree = 1  # Adjust degree as needed (linear = 1, quadratic = 2, etc.)
            coefficients = np.polyfit(cell_voltages, dmm_readings, poly_degree)
            polynomial = np.poly1d(coefficients)

            print(f"Polynomial Fit: {polynomial}")

            # Apply the correction to measured values
            corrected_values = [polynomial(v) for v in cell_voltages]

            # Add corrected values to log
            for i in range(len(data_log)):
                data_log[i]["Corrected Cell Voltage"] = corrected_values[i]

        else:
            print("No valid data for polynomial fitting.")

        # Save data to an Excel file
        df = pd.DataFrame(data_log)
        df.to_excel("measurement_results_with_polynomial.xlsx", index=False)

        print("Data saved to measurement_results_with_polynomial.xlsx")


if __name__ == '__main__':
    main()

And lastly, the one I am having trouble with. But the settings need to be the same on the sketch side for both, well, I guess they really don't, but for ease of use it would be nice, lol.

import pyvisa
import time
import serial
import pandas as pd
import numpy as np
from pymeasure.instruments.keithley import KeithleyDMM6500
import matplotlib.pyplot as plt

# Define the IP address for the Keithley DMM
ip_address_dmm = "XXX.XXX.XXX.XXX"

# Initialize the DMM
dmm = KeithleyDMM6500(f"TCPIP::{ip_address_dmm}::inst0::INSTR")

# Initialize the VISA resource manager
rm = pyvisa.ResourceManager()

# Set DL3021(DL3031A) to DCLOAD
DCLOAD = rm.open_resource("TCPIP::XXX.XXX.XXX.XXX::INSTR")

# Initialize Serial
ser = serial.Serial('COM7', 115200, timeout=2)

# Initialize data storage
data_log = []

def main():
    try:
        time.sleep(4)
        ser.write(b'C')  # 'b'A'' sends the byte 'A' over serial
        DCLOAD.write(':SOUR:CURR:RANG 6')
        
        # Sweep through Current range 4.0 - 5.5A in steps of 0.5A
        for i in range(40, 56, 5):  # Steps of 0.5A (4.0A, 4.5A, ..., 5.5A)
            current = i / 10.00  # Convert to float
            DCLOAD.write(':SOUR:CURR:RANG 6')
            DCLOAD.write(f':SOUR:CURR:LEV:IMM {current}')
            DCLOAD.write(':SOUR:INP:STAT 1')
            time.sleep(2.0)
            dmm_reading = dmm.voltage  # Read voltage from DMM
            scaled_reading = dmm_reading * 2000  # Multiply by 2000
            
            # Read serial data for Icurrent
            try:
                serial_data = ser.readline().decode('utf-8').strip()  # Read and decode
                Icurrent = float(serial_data) if serial_data else None  # Convert to float
            except ValueError:
                Icurrent = None  # Handle non-numeric data
            
            # Store data
            data_log.append({
                "Current (A)": current,
                "Voltage (V)": dmm_reading,
                "Scaled Voltage (V * 2000)": scaled_reading,
                "Icurrent (Serial)": Icurrent
            })
            
            # Print data to the screen
            print(f"Current (A): {current}, Voltage (V): {dmm_reading}, Scaled Voltage (V * 2000): {scaled_reading}, Icurrent (Serial): {Icurrent}")
            
            DCLOAD.write(':SOUR:INP:STAT 0')
            time.sleep(1)
            ser.write(b'C')  # 'b'A'' sends the byte 'A' over serial
            time.sleep(1)
            
        # Sweep through Current range 6.0 - 45.0A in steps of 0.5A
        for i in range(60, 451, 5):  # Steps of 0.5A (6.0A, 6.5A, ..., 45.0A)
            current = i / 10.00  # Convert to float with one decimal place
            DCLOAD.write(':SOUR:CURR:RANG 60')
            DCLOAD.write(f':SOUR:CURR:LEV:IMM {current}')
            DCLOAD.write(':SOUR:INP:STAT 1')
            time.sleep(2.0)
            
            dmm_reading = dmm.voltage  # Read voltage from DMM
            scaled_reading = dmm_reading * 2000  # Multiply by 2000
            
            # Read serial data for Icurrent
            try:
                serial_data = ser.readline().decode('utf-8').strip()  # Read and decode
                Icurrent = float(serial_data) if serial_data else None  # Convert to float
            except ValueError:
                Icurrent = None  # Handle non-numeric data
            
            # Store data
            data_log.append({
                "Current (A)": current,
                "Voltage (V)": dmm_reading,
                "Scaled Voltage (V * 2000)": scaled_reading,
                "Icurrent (Serial)": Icurrent
            })
            
            # Print data to the screen
            print(f"Current (A): {current}, Voltage (V): {dmm_reading}, Scaled Voltage (V * 2000): {scaled_reading}, Icurrent (Serial): {Icurrent}")
            
            DCLOAD.write(':SOUR:INP:STAT 0')
            time.sleep(2)
            ser.write(b'C')  # 'b'A'' sends the byte 'A' over serial
            time.sleep(2)
    
    finally:
        # Ensure the instrument connection is closed
        try:
            DCLOAD.close()
            dmm.close()
            ser.close()
        except Exception as e:
            print(f"Error closing instruments: {e}")
        
        # Save data to an Excel file
        df = pd.DataFrame(data_log)
        df.to_excel("Current_Cal.xlsx", index=False)
        
        # Polynomial Approximation
        fit_polynomial(df)

def fit_polynomial(df):
    """Fit a polynomial to Icurrent vs. Scaled Voltage"""
    df = df.dropna()  # Remove rows with missing values
    
    if len(df) < 2:
        print("Not enough data for polynomial fitting.")
        return
    
    x = df["Scaled Voltage (V * 2000)"].values
    y = df["Icurrent (Serial)"].values
    
    # Fit a second-degree polynomial (quadratic fit)
    coeffs = np.polyfit(x, y, 1)  # Use degree 1 for linear fit, 2 for quadratic, etc.
    poly_eq = np.poly1d(coeffs)
    
    # Generate fit values
    x_fit = np.linspace(min(x), max(x), 100)
    y_fit = poly_eq(x_fit)
    
    # Print polynomial equation
    print("Fitted Polynomial Equation:")
    print(poly_eq)
    
    # Plot data and fit curve
    plt.scatter(x, y, label="Measured Data", color="blue")
    plt.plot(x_fit, y_fit, label=f"Polynomial Fit: {poly_eq}", color="red")
    plt.xlabel("Scaled Voltage (V * 2000)")
    plt.ylabel("Icurrent (Serial)")
    plt.title("Polynomial Fit of Icurrent vs. Scaled Voltage")
    plt.legend()
    plt.grid(True)
    plt.show()

# Run the main function
if __name__ == "__main__":
    main()

Sample output from the cmd line running the python script. When I setup the voltage cal, the same thing happened, it is like the output from the serial is way behind, which is why I added the updates. Funny thing, I originally figured that since the output was behind, I would make the serial update faster, but that made it worse.

Current (A): 4.5, Voltage (V): 0.00225574, Scaled Voltage (V * 2000): 4.51148, Icurrent (Serial): 3.93
Current (A): 5.0, Voltage (V): 0.00250311, Scaled Voltage (V * 2000): 5.00622, Icurrent (Serial): 4.17
Current (A): 5.5, Voltage (V): 0.002752242, Scaled Voltage (V * 2000): 5.504484, Icurrent (Serial): 0.0
Current (A): 6.0, Voltage (V): 0.003002149, Scaled Voltage (V * 2000): 6.004298, Icurrent (Serial): 0.01
Current (A): 6.5, Voltage (V): 0.003253038, Scaled Voltage (V * 2000): 6.506076, Icurrent (Serial): 5.43
Current (A): 7.0, Voltage (V): 0.003504164, Scaled Voltage (V * 2000): 7.008328000000001, Icurrent (Serial): 5.43
Current (A): 7.5, Voltage (V): 0.003754905, Scaled Voltage (V * 2000): 7.50981, Icurrent (Serial): 5.61
Current (A): 8.0, Voltage (V): 0.004005878, Scaled Voltage (V * 2000): 8.011756, Icurrent (Serial): 0.0
Current (A): 8.5, Voltage (V): 0.004256282, Scaled Voltage (V * 2000): 8.512564000000001, Icurrent (Serial): 0.0

Thanks in advance

I've written a small tutorial on interfacing with Python. See Two ways communication between Python3 and Arduino

You must understand that when you post incomplete code, most forum members will assume the problem lies in the part of the code you haven't posted. They will be correct about 50% of the time. So few forum members will try seriously to find it in the part of the code you have posted. If you read the forum guide in the sticky post, there is a warning about this. If you consider your code too long to post, or too secret, then post a shorter version that has everything else removed but still demonstrates the problem.

I did spot an error in your code. It is not the cause of your current problem, but it is a bat habit which you should avoid:

if (serialTime >= serialTime0 + 2500) {

This test will fail, once every 49 days, when millis() rolls over to zero. If you make this very small change, you can avoid the problem:

if (serialTime - serialTime0 >= 2500) {

@J-M-L
Much appreciated, I will go through your tutorial.

@PaulRB
I’m not trying to be secretive, the main part of this code has been posted before and I have received a lot of help on it. But For this part, I really think the serial output was all that was needed. The whole code is over a thousand lines and while commented for me, it would still leave others trying the to work their way through it mostly blind.

That said, I have thought about posting again to help free up some space and make it run faster, but I haven’t got around to it.

Thanks

No problem. I will leave you in the capable hands of the many forum members who have psychic powers.

@J-M-L
Just wanted to say thanks again! The tutorial was great and I definitely used the info to create a buffer and read from the buffer. Also sending a message saying the Arduino was ready is gold! I was using a delay in the python code to wait, but I like your implementation a lot better.

However, all this still didn't fix the issue. I finally fixed it by having the python code send a command to the unit when it wanted the info and then having the unit create that info upon command. This allowed only the reading I wanted into the buffer and everything worked out!

Nice graph of the output:

@PaulRB I did implement your fix on the serial data read time. Although I ended up removing that altogether when I decided to go to explicitly requesting the data. But there was one more part in my overall code that used the same format, so that is fixed.

Thank you both!

1 Like

Well done !

1 Like

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.