Get listenable audio from 2 differents boards microphones (Nicla Voice and Nano RP2040 Connect with Headers)

[mbm974]

Hi,

For a project I need to get audio data from microphones of an Arduino Nicla Voice and an Arduino Nano RP2040 connect.

By using the provided example from the "NDP examples/Record_and_stream" of the Nicla Voice, and by using a python script to retrieve serial data, I managed to get the audio of the Nicla Voice's microphone and can listen to it with audacity in a g722 format (thanks to the AudioTools library encoders).

For the RP2040 microphone, the exemple is the "PDMSerialPlotter" example and stream PDM data through the serial port. The data seem to be raw data and are of the short type (so signed 16 bits ?).

I tried using a python script (with the scipy library) to retrieve and convert the 16bits data to a wav format but get non listenable results (the convertion "works" but it doesn't correspond to sounds played while recording)

I also tried using the G722 encoder from the AudioTools library with the RP2040 PDM data but I get multiple errors while using this code :

/*
  This example reads audio data from the on-board PDM microphones, and prints
  out the samples to the Serial console. The Serial Plotter built into the
  Arduino IDE can be used to plot the audio data (Tools -> Serial Plotter)

  Circuit:
  - Arduino Nano 33 BLE board, or
  - Arduino Nano RP2040 Connect, or
  - Arduino Portenta H7 board plus Portenta Vision Shield, or
  - Arduino Nicla Vision

  This example code is in the public domain.
*/
#include <PDM.h>

#include "AudioTools.h"
#include "AudioCodecs/CodecG722.h"
//#include "AudioI2S/I2SRP2040.h"

G722Encoder encoder;

// default number of output channels
static const char channels = 1;

// default PCM output frequency
static const int frequency = 16000;

// Buffer to read samples into, each sample is 16-bits
short sampleBuffer[512];

// Number of audio samples read
volatile int samplesRead;

void setup() {
  Serial.begin(115200);
  while (!Serial);

  AudioBaseInfo bi;

  bi.sample_rate = 16000;
  bi.channels = 0;

  encoder.setOptions(0);

  encoder.begin();

  // Set the output stream for the encoder to the serial port

  encoder.setOutput(Serial);

  // Configure the data receive callback
  PDM.onReceive(onPDMdata);

  // Optionally set the gain
  // Defaults to 20 on the BLE Sense and 24 on the Portenta Vision Shield
  // PDM.setGain(30);

  // Initialize PDM with:
  // - one channel (mono mode)
  // - a 16 kHz sample rate for the Arduino Nano 33 BLE Sense
  // - a 32 kHz or 64 kHz sample rate for the Arduino Portenta Vision Shield
  if (!PDM.begin(channels, frequency)) {
    Serial.println("Failed to start PDM!");
    while (1);
  }
}

void loop() {
  // Wait for samples to be read
  if (samplesRead) {

    // Print samples to the serial monitor or plotter
    /*for (int i = 0; i < samplesRead; i++) {
      Serial.println(sampleBuffer[i]);
    }*/

    encoder.write(sampleBuffer, samplesRead);

    // Clear the read count
    samplesRead = 0;
    
  }
}

/**
 * Callback function to process the data from the PDM microphone.
 * NOTE: This callback is executed as part of an ISR.
 * Therefore using `Serial` to print messages inside this function isn't supported.
 * */
void onPDMdata() {
  // Query the number of available bytes
  int bytesAvailable = PDM.available();

  // Read into the sample buffer
  PDM.read(sampleBuffer, bytesAvailable);

  // 16-bit, 2 bytes per sample
  samplesRead = bytesAvailable / 2;
}

ATTENTION: la bibliothèque I2S prétend être exécutable sur la (ou les) architecture(s) rp2040 et peut être incompatible avec votre carte actuelle qui s'exécute sur mbed_nano.
ATTENTION: la bibliothèque AudioBufferManager prétend être exécutable sur la (ou les) architecture(s) rp2040 et peut être incompatible avec votre carte actuelle qui s'exécute sur mbed_nano.
In file included from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioI2S/I2SRP2040.h:8:0,
from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioI2S/I2SStream.h:16,
from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioTools.h:60,
from C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:16:
C:\Users\User\Documents\Arduino\libraries\I2S\src/I2S.h:137:5: error: 'PIOProgram' does not name a type
PIOProgram _i2s;
^~~~~~~~~~
C:\Users\User\Documents\Arduino\libraries\I2S\src/I2S.h:138:5: error: 'PIO' does not name a type; did you mean 'EIO'?
PIO _pio;
^~~
EIO
In file included from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/AudioPWM.h:6:0,
from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioTools.h:61,
from C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:16:
C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/PWMAudioRP2040.h: In member function 'void audio_tools::PWMDriverRP2040::setupPWMPin(pwm_config&, audio_tools::PicoChannelOut&)':
C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/PWMAudioRP2040.h:131:37: error: invalid conversion from 'pwm_config' to 'uint {aka unsigned int}' [-fpermissive]
pwm_init(pinInfo.slice, &cfg, true);
^~~~
C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/PWMAudioRP2040.h:131:47: error: cannot convert 'bool' to 'pwm_config*' for argument '3' to 'void pwm_init(uint, uint, pwm_config*, bool)'
pwm_init(pinInfo.slice, &cfg, true);
^
In file included from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/AudioPWM.h:7:0,
from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioTools.h:61,
from C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:16:
C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/PWMAudioMBED.h: At global scope:
C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/PWMAudioMBED.h:17:32: error: conflicting declaration 'using PWMDriver = class audio_tools::PWMDriverMBED'
using PWMDriver = PWMDriverMBED;
^
In file included from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/AudioPWM.h:6:0,
from C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioTools.h:61,
from C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:16:
C:\Users\User\Documents\Arduino\libraries\audio-tools\src/AudioPWM/PWMAudioRP2040.h:21:34: note: previous declaration as 'using PWMDriver = class audio_tools::PWMDriverRP2040'
using PWMDriver = PWMDriverRP2040;
^
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:34:12: error: 'void audio_tools::setup()' conflicts with a previous declaration
void setup() {
^
In file included from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\variants\NANO_RP2040_CONNECT/pinmode_arduino.h:30:0,
from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/Arduino.h:26,
from C:\Users\User\AppData\Local\Temp\arduino\sketches\8BD11F1F7567A81BAC09DF15E8AE4FBC\sketch\PDMspG722Encoder.ino.cpp:1:
C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/api/Common.h:117:6: note: previous declaration 'void setup()'
void setup(void);
^~~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:68:11: error: 'void audio_tools::loop()' conflicts with a previous declaration
void loop() {
^
In file included from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\variants\NANO_RP2040_CONNECT/pinmode_arduino.h:30:0,
from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/Arduino.h:26,
from C:\Users\User\AppData\Local\Temp\arduino\sketches\8BD11F1F7567A81BAC09DF15E8AE4FBC\sketch\PDMspG722Encoder.ino.cpp:1:
C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/api/Common.h:118:6: note: previous declaration 'void loop()'
void loop(void);
^~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino: In function 'void audio_tools::setup()':
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:34:12: error: 'void audio_tools::setup()' conflicts with a previous declaration
void setup() {
^
In file included from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\variants\NANO_RP2040_CONNECT/pinmode_arduino.h:30:0,
from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/Arduino.h:26,
from C:\Users\User\AppData\Local\Temp\arduino\sketches\8BD11F1F7567A81BAC09DF15E8AE4FBC\sketch\PDMspG722Encoder.ino.cpp:1:
C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/api/Common.h:117:6: note: previous declaration 'void setup()'
void setup(void);
^~~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:52:7: error: request for member 'onReceive' in 'PDM', which is of non-class type 'audio_tools::I2SSignalType'
PDM.onReceive(onPDMdata);
^~~~~~~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:62:12: error: request for member 'begin' in 'PDM', which is of non-class type 'audio_tools::I2SSignalType'
if (!PDM.begin(channels, frequency)) {
^~~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino: In function 'void audio_tools::loop()':
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:68:11: error: 'void audio_tools::loop()' conflicts with a previous declaration
void loop() {
^
In file included from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\variants\NANO_RP2040_CONNECT/pinmode_arduino.h:30:0,
from C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/Arduino.h:26,
from C:\Users\User\AppData\Local\Temp\arduino\sketches\8BD11F1F7567A81BAC09DF15E8AE4FBC\sketch\PDMspG722Encoder.ino.cpp:1:
C:\Users\User\AppData\Local\Arduino15\packages\arduino\hardware\mbed_nano\4.0.2\cores\arduino/api/Common.h:118:6: note: previous declaration 'void loop()'
void loop(void);
^~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino: In function 'void audio_tools::onPDMdata()':
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:90:16: error: 'void audio_tools::onPDMdata()' conflicts with a previous declaration
void onPDMdata() {
^
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:90:6: note: previous declaration 'void audio_tools::onPDMdata()'
void onPDMdata() {
^~~~~~~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:92:28: error: request for member 'available' in 'PDM', which is of non-class type 'audio_tools::I2SSignalType'
int bytesAvailable = PDM.available();
^~~~~~~~~
C:\Users\User\Documents\Arduino\PDMSerialPlotter\PDMspG722Encoder\PDMspG722Encoder.ino:95:7: error: request for member 'read' in 'PDM', which is of non-class type 'audio_tools::I2SSignalType'
PDM.read(sampleBuffer, bytesAvailable);
^~~~

exit status 1

Compilation error: 'void audio_tools::setup()' conflicts with a previous declaration

In the first error, I looked where the PIOProgram class should be located in the GitHub repository of the Arduino-pico repository and it comes from one of the cores files (this one).

How should I used these core files?

Is there a better/simpler way to retrieve and listen to audio data from the Nano RP2040 Connect board ?

Ideally, I would like to only use raw data from both microphones (uint8_t data from the NiclaVoice and short data from the Nano RP2040) and use externals tools (like a python script) to encode the data in a listenable audio format. But my tries don't seem to work.

If you can help me with 1) retrieving encoded data from the RP2040 microphone or 2) encode and listen to raw data from the Nicla Voice and RP2040 microphone I would be very grateful.

Thanks in advance.

[awneil]

@mbm974

the example is the "PDMSerialPlotter" example and stream PDM data through the serial port

PDM = Pulse Density Modulation - might help in searching for how to decode ...

[mbm974]

Thanks for the hint.

I looked through the build-in PDM library but nothing is available for decoding from PDM to PCM.

I saw a video tutorial that described the steps to decode PDM values to PCM, do you know if its the appropriate way to do it ? If there are any arduino library existing to do it ?

I'm trying to implement it in python for the moment but if you have any suggestions that would be great.

The process i'm trying to implement for the moment is : 1. apply a low pass FIR filter over the PDM data, 2. add decimation to adjust the PCM sample rate, 3. apply a high pass filter if there is a DC offset

Thanks in advance

[mbm974]

And additional question : why the AudioTools library has no trouble encoding PDM data from NiclaVoice microphone but there are when using the PDM data from the RP2040 microphone ? Maybe I'm not using it right in my provided code.

[mbm974]

Can someone provide me with feedback on my code for converting PDM data to PCM please ?

from scipy.signal import firwin, kaiserord

signal_sample_rate = 16000
nyq_rate = signal_sample_rate / 2 #(sample_rate/2)
# sig.firwin()

# The desired width of the transition from pass to stop,
# relative to the Nyquist rate.  We'll design the filter
# with a 5 Hz transition width.
width = 2000/nyq_rate

# The desired attenuation in the stop band, in dB.
ripple_db = 30.0

#The cutoff frequency of the filter.
cutoffhz = 100.0/nyq_rate

# Compute the order and Kaiser parameter for the FIR filter.
window_size, beta = kaiserord(ripple_db, width)
print(window_size)
print(beta)

#list with a weight for each window_size value, weights decided by a FIR filter
weight = firwin(numtaps = window_size, cutoff = cutoffhz, window = ('kaiser',beta))
print(weight)
lowPass = []
decimation = 1
gain = 20
new_sample_rate = signal_sample_rate / decimation
for i in range(len(signalBit)-window_size):
    if(i%decimation == 0):
        add = 0
        for j in range(window_size):
            add = add + signalBit[i+j] * weight[j]
        mean = add/window_size * gain
        lowPass.append(mean)

I would like to keep the same frequency as the original PCM data so I decide not to use decimation. For the high pass filter, because I don't have a working conversion so far, I can't know if I need to use it for the moment.

Thanks in advance

[ptillisch]

@mbm974

error: invalid conversion from 'pwm_config*' to 'uint {aka unsigned int}' [-fpermissive]
pwm_init(pinInfo.slice, &cfg, true);
^~~~

In case you are still interested in the subject, there is some information about the problem here:

Library RP2040_pwm don't compile

Hi @glenton. I investigated and found that the developers of the "Arduino Mbed OS Nano Boards" platform apply a patch to the "Pico C SDK" of the RP2040 microcontroller that makes an undocumented breaking change to the API: That change to the Pico SDK API breaks the "RP2040_PWM" library. I'll provide three options for fixing/working around this problem. You can pick whichever one of the three that is most convenient to you. A. Use the "Raspberry Pi Pico/RP2040" boards platform The "Raspberry…

That is about a different library from the "audio-tools" library you are using, but the information there about the cause and the solution is applicable to the audio-tools library as well.

[DVDdoug]

I don't know anything about these libraries but I know a lot about audio.

@mbm974

I would like to keep the same frequency as the original PCM data so I decide not to use decimation.

Same as PDM?

You can't. PDM needs a higher sample-clock rate. PDM is one-bit at a time so it's not a "traditional sample". A normal (PCM) sample represents the wave amplitude at one instant in time. The higher the bit-depth, the higher the amplitude resolution. You "connect the dots" (at the correct sample-rate) and smooth, and you have analog again.