HELP Issue in transferring Binary data from DUE to Python

[li_a]

Hi everyone, I’m new to Arduino and electronics, and this is my first post. I've had success with previous projects thanks to this forum, but I’m currently stuck on a new one.

The project: I'm measuring vibrations (up to 20kHz) using a Joy-it piezoelectric disk sensor (SEN-VIB01) and an Arduino Due. I need to record the signal for a few minutes at a sampling rate of 50kHz.

The data is converted to binary and recorded in Python (using Spyder), but the results seem off. I’m getting constant jumps between 0 and 60,000.

To check the system, I record data without touching the piezo, then press it to see if I can detect the impulse in the time domain.

Note: Reducing the baud rate to 115200 and sampling rate to 5kHz gives better results (I can detect the impulse), but the values are still higher than expected.

Attached is a picture of my setup (the USB connects to my laptop):

Here is my arduino code:

#include <Arduino.h>

#define BUFFER_SIZE 20000  // Adjust buffer size as needed
volatile uint16_t bufferIndex = 0;  // Index to track buffer position
uint8_t buffer[BUFFER_SIZE];        // Buffer to hold sensor values
volatile bool bufferFull = false;   // Flag to indicate buffer is full

void setup() {
  // Initialize Serial communication at 921600 baud rate
  Serial.begin(921600);

  // Configure the ADC for fast readings
  pmc_enable_periph_clk(ID_ADC);        // Enable the ADC clock
  ADC->ADC_MR |= ADC_MR_FREERUN_OFF;    // Disable free-run mode (manual trigger mode)
  ADC->ADC_MR |= ADC_MR_PRESCAL(1);     // Set ADC clock prescaler for faster readings
  ADC->ADC_CHER |= ADC_CHER_CH0;        // Enable channel 0 (A0 pin)

  // Set up Timer Counter (TC0, Channel 0) for 50kHz (20 µs interval)
  pmc_set_writeprotect(false);          // Disable write protection
  pmc_enable_periph_clk(ID_TC0);        // Enable clock for TC0

  // Configure Timer Counter
  TC_Configure(TC0, 0, TC_CMR_TCCLKS_TIMER_CLOCK1 | TC_CMR_WAVE | TC_CMR_WAVSEL_UP_RC);
  uint32_t rc = 840;                    // Set compare value for 50kHz (84MHz / 2 / 50kHz = 840)
  TC_SetRC(TC0, 0, rc);                 // Set the compare match register
  TC_Start(TC0, 0);                     // Start the timer

  // Enable the interrupt for the TC0 handler (Timer Counter 0)
  TC0->TC_CHANNEL[0].TC_IER = TC_IER_CPCS;  // Enable interrupt on RC compare
  NVIC_EnableIRQ(TC0_IRQn);                 // Enable TC0 interrupt in the NVIC
}

void loop() {
  // If buffer is full, send it via Serial
  if (bufferFull) {
    bufferFull = false;
    Serial.write(buffer, BUFFER_SIZE);  // Send the buffer
  }
}

// Timer Counter 0 Interrupt Service Routine (ISR)
void TC0_Handler() {
  TC_GetStatus(TC0, 0);  // Clear the interrupt flag

  // Start an ADC conversion
  ADC->ADC_CR = ADC_CR_START;

  // Wait until the conversion is complete
  while (!(ADC->ADC_ISR & ADC_ISR_EOC0)) {
    // Do nothing, wait for conversion to complete
  }

  // Read the ADC value from channel 0 (A0 pin)
  uint16_t sensorValue = ADC->ADC_CDR[0];

  // Store high and low bytes in the buffer
  buffer[bufferIndex++] = (sensorValue >> 8) & 0xFF;  // High byte
  buffer[bufferIndex++] = sensorValue & 0xFF;         // Low byte

  // Check if buffer is full
  if (bufferIndex >= BUFFER_SIZE) {
    bufferIndex = 0;        // Reset buffer index
    bufferFull = true;      // Set flag to indicate buffer is ready to send
  }

Here is my Python code:

# -*- coding: utf-8 -*-
"""
"""

import serial
import time
import struct
import os

baudr = 921600 # adjust baud rate to file

# Open the serial connection (adjust COM port and baud rate)
ser = serial.Serial('COM6', baudr, timeout=1)
time.sleep(2)  # Give some time for the connection to establish
# Open a binary file to store the data
with open('data.bin', 'wb') as file:
    try:
        while True:
            # Read 2 bytes (16 bits) from the serial port
            raw_data = ser.read(2)
            if len(raw_data) == 2:
                # Write binary data to the file
                file.write(raw_data)

                # Optional: Convert to 16-bit integer and print to console
                sensor_value = struct.unpack('>H', raw_data)[0]  # Big-endian 16-bit int
                print(sensor_value) 
                #voltage = sensor_value *(3.3/1023)
                #print(voltage)
                # Print raw byte data
                print(f"Raw data (bytes): {raw_data}")
    except KeyboardInterrupt:
        print("Data collection stopped.")
    finally:
        # Ensure the serial port is closed properly
        ser.close()
        print("Serial port closed.")
        
        print(os.getcwd())

The output in Python looks like that :

Raw data (bytes): b'\x7f\xd0'
34567
Raw data (bytes): b'\x87\x07'
49040
Raw data (bytes): b'\xbf\x90'
34687
Raw data (bytes): b'\x87\x7f'
36999
Raw data (bytes): b'\x90\x87'
32720
Raw data (bytes): b'\x7f\xd0'
1927
Raw data (bytes): b'\x07\x87'

and here is a graph showing the absolute nonsense I get:

I would really appreciate it if any of you could help me figure out whats wrong with my setup / program.

Thank you.

[van_der_decken]

You say you're new to Arduino and electronics, and yet your sketch is using register level programming. Your sketch was created by ChatGPT or one of its cousins, wasn't it? And you don't have the level of knowledge to know whether or not it's made up something that looks plausible, but has no hope of working, am I right?

[li_a]

of course I used some ChatGPT, but only in increment and going regularly back and forth and testing. The main functions used, I adapted them from other solutions found on this Forum. You would not get this code from chatgpt by simply writting the project description as I did above. It is really moving to binary that has been a bit of struggle.

[van_der_decken]

Good luck with your project. I suggest asking ChatGPT what is wrong with it. Good bye.

[kolaha]

if i want read from serial, i need ask if it ready.
DUE is able to write file by himself so you can open it in Calc/Excel

[jremington]

@li_a

I’m getting constant jumps between 0 and 60,000.

This is a code problem, due to overflow of a 16 bit integer and/or confusion of signed versus unsigned 16 bit integer variables. Typical for code produced by chatGPT, which does not "know" the rules.

If you don't know the rules either, you are in a very poor position to recognize the errors in code spewed out by a bot.

[Paul_KD7HB]

@li_a

The data is converted to binary and recorded in Python (using Spyder), but the results seem off. I’m getting constant jumps between 0 and 60,000.

Here is your basic problem: EVERYTHING in your Arduino is binary. All other aspects are how that binary is looked at and processed. ASCII is one way of looking at binary. HEX is another way of looking at binary. Neither changes the binary, only represents it differently.
Your original data is already binary. You may do anything you want with the measurements, but they will always be binary. You can convert the binary to some other coding system, but it is still binary.
Your programming must keep track of how you are currently looking at the binary values, so the calculations make sense.

[horace]

if you wish to sample 50000samples/second

1. 16 bit data would be 100000bytes/second
2. 12 bit data you could bit shift to approximately 75000 bytes/second

using serial at 115200 baud you can transfer approximately 10000bytes/second at 921600baud 90000bytes/second

consider using a esp32 communicating using using Bluetooth/BLE, WiFi UDP or even Ethernet

to sample 50000samples/second you probably need to use DMA

you may need to move to more powerful processors, e.g. Texas DSP

[li_a]

My understanding was that the process of using the Due to write signals in excel in a human readable format would be too slow for my set up. or do you think it is feasible ?

[li_a]

Thank you for you responses @Paul_KD7HB and @jremington. I understand that all data communication is done in binary. In many of the examples I read on transferring data from arduino to python, the described process was to send data in a 'human readable format' which I understand is not suitable in my case (ref this example: Fastest way to read data into Python). I thought I was close to getting it right because of the plausible output I had at lower frequency and lower baud rate. but I will try again

[li_a]

Thank you @horace I'll have a look at that too

[kolaha]

no need to write human readable format on the fly, but maybe DUE is fast enough

[horace]

assuming you are transferring 16bit integers as two bytes over serial when you receive two bytes how do you know which is the MSB (Most Significant Byte) and LSB (Least Significant Bytes)

consider the following program which transmits a uint16_t as two bytes

// write unsigned 16bit binary values to Serial as two bytes

void setup() {
  Serial.begin(115200);
}

void loop() {
  static uint16_t data = 0;
  Serial.write((data >> 8) & 0xff);   // write MSB byte
  Serial.write(data & 0xff);          // write LSB byte
  data += 50;
  delay(500);
}

and a Python program which reads two bytes to form a 16bit integer

#read binary 16bit unsigned data from Serial as two bytes checking for lost data
# will give an error on startup while it syncs
#  and when local int overflows 65535

import serial
import time
import math
ser = serial.Serial('COM11',115200,timeout=1)
time.sleep(1)   
data=0
errors=0
ser.flush()
while True:
    if ser.in_waiting > 0:
        bytein = ser.read(2)
        datain=int.from_bytes(bytein,"big")
        #print(datain)
        if data != datain:
            errors=errors+1
            print("error received ", datain," should be ", data," errors ", errors)
            data=datain
        else:
            print('received=', datain," errors ", errors)
        data=data+50

it may work receiving the bytes in the correct order MSB then LSB

error received  2700  should be  0  errors  1
received= 2750  errors  1
received= 2800  errors  1
received= 2850  errors  1
received= 2900  errors  1
received= 2950  errors  1
received= 3000  errors  1
received= 3050  errors  1
received= 3100  errors  1
received= 3150  errors  1
received= 3200  errors  1
received= 3250  errors  1
received= 3300  errors  1
received= 3350  errors  1
received= 3400  errors  1
received= 3450  errors  1
received= 3500  errors  1
received= 3550  errors  1
received= 3600  errors  1

-------------------------------------------------------

received= 17950  errors  1
received= 18000  errors  1
received= 18050  errors  1
received= 18100  errors  1
received= 18150  errors  1
received= 18200  errors  1
received= 18250  errors  1
received= 18300  errors  1
received= 18350  errors  1
received= 18400  errors  1
received= 18450  errors  1
received= 18500  errors  1
received= 18550  errors  1
received= 18600  errors  1
received= 18650  errors  1
received= 18700  errors  1
received= 18750  errors  1
received= 18800  errors  1
received= 18850  errors  1
received= 18900  errors  1
received= 18950  errors  1
received= 19000  errors  1
received= 19050  errors  1
received= 19100  errors  1
received= 19150  errors  1
received= 19200  errors  1


--------------------------------------------------

received= 65050  errors  1
received= 65100  errors  1
received= 65150  errors  1
received= 65200  errors  1
received= 65250  errors  1
received= 65300  errors  1
received= 65350  errors  1
received= 65400  errors  1
received= 65450  errors  1
received= 65500  errors  1
error received  14  should be  65550  errors  2
received= 64  errors  2
received= 114  errors  2
received= 164  errors  2
received= 214  errors  2
received= 264  errors  2
received= 314  errors  2
received= 364  errors  2
received= 414  errors  2
received= 464  errors  2
received= 514  errors  2
received= 564  errors  2

or maybe not

error received  64001  should be  51250  errors  171
error received  11265  should be  64051  errors  172
error received  24065  should be  11315  errors  173
error received  36865  should be  24115  errors  174
error received  49665  should be  36915  errors  175
error received  62466  should be  49715  errors  176
error received  9730  should be  62516  errors  177
error received  22530  should be  9780  errors  178
error received  35330  should be  22580  errors  179
error received  48130  should be  35380  errors  180
error received  60931  should be  48180  errors  181
error received  8195  should be  60981  errors  182
error received  20995  should be  8245  errors  183
error received  33795  should be  21045  errors  184
error received  46595  should be  33845  errors  185
error received  59396  should be  46645  errors  186
error received  6660  should be  59446  errors  187
error received  19460  should be  6710  errors  188
error received  32260  should be  19510  errors  189
error received  45060  should be  32310  errors  190
error received  57861  should be  45110  errors  191
error received  5125  should be  57911  errors  192

generally I would transmit such data in a frame with start and end sequences, CRC check and using byte stuffing