Storing multiple data from arduino into a CSV file using python

Hi totally newbie to python here.
I was following this tutorial on how to get data from arduino and saving it as a csv file https://engineersportal.com/blog/2018/2/...g-pyserial

I would like to know how to log multiple sensor values. I know it should be something to do with data parsing.
Here is the code which I use to get the value of just one sensor.

import serial
import time
import csv
 
ser = serial.Serial('COM3')
ser.flushInput()
 
while True:
    try:
        ser_bytes = ser.readline()
        decoded_bytes = float(ser_bytes[0:len(ser_bytes)-2].decode("utf-8"))
        print(decoded_bytes)
        with open("test_data.csv","a", newline='') as f:
            writer = csv.writer(f,delimiter=",")
            writer.writerow([time.strftime("%H : %M : %S"),decoded_bytes])
    except:
        print("Keyboard Interrupt")
        break

Yami89:
Hi totally newbie to python here.

This is a Forum about programming Arduinos. For help with a Python program you need to ask on a Python Forum.

There must be dozens or hundreds of online tutorials about CSV with Python.

...R

Let the Arduino send the comma separated sensor readings as text using Serial.print / Serial.println

Your python code only needs to read received characters and store them in a file.

Robin2:
This is a Forum about programming Arduinos. For help with a Python program you need to ask on a Python Forum.

There must be dozens or hundreds of online tutorials about CSV with Python.

...R

Yep tried some python forums no luck :frowning: ... I'm going through some tutorials at the moment.

sterretje:
Let the Arduino send the comma separated sensor readings as text using Serial.print / Serial.println

Your python code only needs to read received characters and store them in a file.

Ahh make sense, I did try it but all I get is the python program stops and displays that a keyboard Interrupt was received.
Would really appreciate if you could direct me towards some tutorials examples etc. Currently having no luck, either it doesn't work or the code is too complex. The code I'm using I could get grips to it I know it just need a bit of tweaking.

Thanks so much again :slight_smile:
Y

This Python - Arduino demo may be of interest. I had been assuming you already had reliable communication between your Arduino and Python.

...R

I have no serious experience with python (I had to steal parts from your code and look up the rest) but with a little digging I came up with following (non-perfect) code

import serial
#import keyboard

try:
    ser = serial.Serial('COM4', baudrate=57600)
    ser.flushInput()

    while True:
        ser_bytes = ser.readline()
        print(ser_bytes)
        file = open("testfile.csv", "a")
        file.write(str(ser_bytes))
        file.close()
        
 #       if keyboard.is_pressed('esc'):
 #           break;
    ser.close
except:
    print("Unexpected error:", sys.exc_info()[0])
    print("Unexpected error:", sys.exc_info()[1])

The keyboard.is_pressed was an attempt to try to be able to interrupt the loop but it kept throwing an error.

Note that you might get rubbish with the first data (I suspect a remains of the bootloader); look at robin's example how he solved it.

The keyboard issue was caused by an incorrect installation of the library.

Nice to learn something new :slight_smile: Thanks for that 8)

I came up with the below python code

#run with C:\Python34\py.exe serial_test.py

import sys
import serial
import keyboard

print("modules start")
print(keyboard.__file__)
print(serial.__file__)
print("modules end")

try:
    # open the serial port; only do this once as most arduinos reset when the serial port is opened
    ser = serial.Serial('COM4', baudrate=57600)
    # e.g. remove remains of Arduino bootloader or old data while the application was not running
    # not sure yet if it's 100% reliable; robin's approach is probably safer
    ser.flushInput()

    while True:
        # check if bytes received
        numBytes = ser.inWaiting()
        if(numBytes > 0):
            serBytes = ser.readline()
            print(serBytes)
            # open file for binary (!) appending; not using binary results in
            # 1) error telling you 'must be str, not bytes'
            # 2) convering using str(ser_bytes) results in unwanted quotation marks in the file (as shown in the result of above print)
            file = open('testfile.csv', 'ab')
            file.write(serBytes)
            file.close()

            # check if <esc> was pressed; stop if so
        if keyboard.is_pressed('esc'):
            break;

    # close serial port
    ser.close
except:
    print("Unexpected error:", sys.exc_info()[0])
    print("Unexpected error:", sys.exc_info()[1])

    # maybe todo: close serial port; this might need a little rework of above code moving 'ser = serial.Serial('COM4', baudrate=57600)' to outside the try

The first line is for myself so I don’t have to learn each time how to execute a script

And tested with the below Arduino code

void setup()
{
  Serial.begin(57600);
}

void loop()
{
  int x, y;
  char text[] = "Some text, this contains a comma, an escaped double quote \"\" and line ending\r\n before this";
  x = random();
  y = random();

  // simple prints
  Serial.print(x);
  Serial.print(",");
  Serial.print(y);
  Serial.print(",");

  // precautions when printing text; embed in double quotes
  Serial.print("\"");
  Serial.print(text);
  Serial.print("\"");

  // end of csv record
  Serial.print("\n");

  delay(2000);
}

The way text is printed is not perfect; one should write a function that does the escaping and embedding in double quotes. It was merely added to show possible precautions that you have to take when embedding special characters like commas and double quotes) in text.

The result

16807,15089,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-21287,3114,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-18558,-9528,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-28968,2558,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
12099,1101,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-26472,15445,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
16807,15089,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-21287,3114,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-18558,-9528,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-28968,2558,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
12099,1101,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"
-26472,15445,"Some text, this contains a comma, an escaped double quote "" and line ending
 before this"

If you open this in a text editor, you will see that each record consists of two lines.
If you open this in e.g. Excel (or equivalent), you will probably have to adjust the height of the rows to see that each record consists of two lines and that there is only one double quote in the text. You might have to adjust the width of the text column

I leave implementing command line parameters to select a com port, baudrate and the likes and the output filename to you