I found the .h file named "src/AudioI2S/I2SRP2040-MBED.h"
I thought that maybe this would be compatible with the board. I am compiling in arduino IDE. I get the following error:
fatal error: FreeRTOS.h: No such file or directory, include "FreeRTOS.h"
I did at one point download the FreeRTOS. But I could see it was only supporting AVR architecture, as mentioned this is mbed_os. I was compiling the simple star wars example in your readme file. Is there another way or a different type of RTOS I should be using for my board?
HI, I am trying to achieve the PWM or I2S in the mbed architecture. As that is also where the voice recognition module lies. Within supported architecture here, in library properties:
Hey, thanks for reply. I am starting to understand there are better choices in MCU. However this nano rp2040 presents an interesting prospect with offline voice recognition capabilities. Hence the purchase.
I know I could use esp32 microphone and then post process maybe in the cloud or something using tensorflow. I was also constrained as this need to be in a really small package.
Also im in a country that takes like 2 weeks to get parts. Not to go on a long rant of why I chose lol. But yea im trying to utilise what I have which I think is enough.
I've thought through trying to use the rpi sdk as the audio I2S and this side works. But the voice recognition works on mbed_os so its super frustrating.
I just found this lib. And it could be the answer to having I2S work on the mbed_os.
Hi I now have the I2S playing audio from an SD card and to the speakers through a MAX98357a DAC/amp. This is using MBED_OS as opposed to the raspberry pi development environment. using the following lib:
My issue now is that the I2S.begin does not intialise. Even when I take out the while loop and put an else print 'failed'. It seems to get stuck. the Arduino nano rp2040 just remains flashing orange. And it blocks the cyberon model from going to active listening state.
My thoughts are that there may be a resource conflict between Cyberon model using PCM on the mic and the I2S being used for audio output. Can any one shed any light on this?
#include <Arduino.h>
#include <DSpotterSDK_MakerHL.h>
#include <LED_Control.h>
#include <I2S.h>
#include <SD.h>
#include <SPI.h>
// The DSpotter License Data.
#include "CybLicense.h"
#define DSPOTTER_LICENSE g_lpdwLicense
// The DSpotter Keyword Model Data.
#if defined(TARGET_ARDUINO_NANO33BLE) || defined(TARGET_PORTENTA_H7) || defined(TARGET_NICLA_VISION)
// For ARDUINO_NANO33BLE and PORTENTA_H7
#include "Model_L1.h" // The packed level one model file.
// For NANO_RP2040_CONNECT
#elif defined(TARGET_NANO_RP2040_CONNECT)
#include "Model_L0.h" // The packed level zero model file.
#endif
#define DSPOTTER_MODEL g_lpdwModel
// The VR engine object. Only can exist one, otherwise not worked.
static DSpotterSDKHL g_oDSpotterSDKHL;
const int chipSelect = 29; // SD card chip select pin
File wavFile;
// Callback function for VR engine
void VRCallback(int nFlag, int nID, int nScore, int nSG, int nEnergy)
{
if (nFlag==DSpotterSDKHL::InitSuccess)
{
//ToDo
}
else if (nFlag==DSpotterSDKHL::GetResult)
{
/*
When getting an recognition result,
the following index and scores are also return to the VRCallback function:
nID The result command id
nScore nScore is used to evaluate how good or bad the result is.
The higher the score, the more similar the voice and the result command are.
nSG nSG is the gap between the voice and non-command (Silence/Garbage) models.
The higher the score, the less similar the voice and non-command (Silence/Garbage) models are.
nEnergy nEnergy is the voice energy level.
The higher the score, the louder the voice.
*/
//ToDo
}
else if (nFlag==DSpotterSDKHL::ChangeStage)
{
switch(nID)
{
case DSpotterSDKHL::TriggerStage:
LED_RGB_Off();
LED_BUILTIN_Off();
break;
case DSpotterSDKHL::CommandStage:
LED_BUILTIN_On();
break;
default:
break;
}
}
else if (nFlag==DSpotterSDKHL::GetError)
{
if (nID == DSpotterSDKHL::LicenseFailed)
{
//Serial.print("DSpotter license failed! The serial number of your device is ");
//Serial.println(DSpotterSDKHL::GetSerialNumber());
}
g_oDSpotterSDKHL.Release();
while(1);//hang loop
}
else if (nFlag == DSpotterSDKHL::LostRecordFrame)
{
//ToDo
}
}
void setup()
{
// Init LED control
LED_Init_All();
// Init Serial output for show debug info
Serial.begin(9600);
while(!Serial);
DSpotterSDKHL::ShowDebugInfo(true);
// Init VR engine & Audio
if (g_oDSpotterSDKHL.Init(DSPOTTER_LICENSE, sizeof(DSPOTTER_LICENSE), DSPOTTER_MODEL, VRCallback) != DSpotterSDKHL::Success)
return;
// Initialize SD card
if (!SD.begin(chipSelect)) {
Serial.println("Failed to initialize SD card!");
} else {
Serial.println("Initialised SD card");
}
// Open the WAV file
wavFile = SD.open("lords.wav");
if (!wavFile) {
Serial.println("Failed to open WAV file!");
} else {
Serial.println("Opened WAV file");
}
// Skip WAV header (44 bytes for standard WAV files)
wavFile.seek(44);
// Start I2S with a sample rate of 44100 Hz (CD quality), 16 bits per sample
if (!I2S.begin(22500)) {
Serial.println("Failed to initialize I2S!");
while (1);
}
}
void loop()
{
// Do VR
g_oDSpotterSDKHL.DoVR();
}
Your two or more topics on the same or similar subject have been merged.
Please do not duplicate your questions as doing so wastes the time and effort of the volunteers trying to help you as they are then answering the same thing in different places.
Please create one topic only for your question and choose the forum category carefully. If you have multiple questions about the same project then please ask your questions in the one topic as the answers to one question provide useful context for the others, and also you won’t have to keep explaining your project repeatedly.
Repeated duplicate posting could result in a temporary or permanent ban from the forum.
Could you take a few moments to Learn How To Use The Forum
It will help you get the best out of the forum in the future.
Hi stand alone. This code works. which is I2S_rp2040 library mentioned above.
The reason for using 22500 opposed to standard 44khz is because it was running at double speed although i changed mp3 to WAV at 44khz. The reason for which I am not clear on just yet. I also tried 44khz with the Cyberon model integration. And it did not make a difference to the 'I2S.begin' hang. Just to confirm the script below at selected sample rate works. Its when I use with the model the begin gets stuck in a hang state.
#include <I2S.h>
#include <SD.h>
#include <SPI.h>
const int chipSelect = 29; // SD card chip select pin
File wavFile;
void setup() {
Serial.begin(115200);
// Initialize SD card
if (!SD.begin(chipSelect)) {
Serial.println("Failed to initialize SD card!");
while (1);
}
// Open the WAV file
wavFile = SD.open("lords.wav");
if (!wavFile) {
Serial.println("Failed to open WAV file!");
while (1);
}
// Skip WAV header (44 bytes for standard WAV files)
wavFile.seek(44);
// Start I2S with a sample rate of 44100 Hz (CD quality), 16 bits per sample
if (!I2S.begin(22500)) {
Serial.println("Failed to initialize I2S!");
while (1);
}
}
void loop() {
// Read and send samples to I2S
while (wavFile.available()) {
int16_t sample = wavFile.read(); // Read low byte
sample |= wavFile.read() << 8; // Read high byte
I2S.write(sample); // Send sample to I2S (left channel)
I2S.write(sample); // Send same sample to I2S (right channel)
}
// Close file when done
wavFile.close();
while (1);
}
It’s a great point, I thought maybe the Arduino.h would conflict with the i2s.h and the way the pins are defined. So i commented Arduino.h out compiled and flashed.
Still wouldn’t work. As side note not sure why they put Arduino.h in the sketch. Maybe if I used platformio or something like this.
Do you have any advice of how I can dig deeper into the libraries which are used for cyberon and the pin mapping they are using. To see if I have a conflict?
First check if you are running out of memory. I2S begin() probably allocates a large RAM buffer and, I assume, so does the model library. The order in which you initialize the objects may make a difference, and the return values of .Init and .begin may be informative.
There are various free memory functions, so you can check memory status at any place in the code, but I'm not familiar with what is available for the RP2040.
Then go through the library source code and note all pin assignments.
Edit: check whether the voice model library takes over the I2S port.
Those would be unsigned positive integers, misinterpreted by Serial.print as signed, and are changing in the expected direction for 8 kB buffer allocation.
Hey that makes sense now. I found a lib that had a little script in it. And it seems to be giving me realistic values now. I tried a simple I2S begin and then reading before and after the memory. I got the following results:
15:49:42.042 -> Free memory before I2S init: 88780
15:49:42.299 -> Free memory after I2S init: 86030
I then put this memory read back into the main voice recognition model. I got the following reuslts:
19:39:35.297 -> Free memory start of init: 30480
19:39:35.378 -> Free memory before I2S init: 30480
19:39:36.233 -> Free memory before Cyberon model init: 30350
Note ! I actually commented out the I2S init because it got stuck in hang and crashed the Arduino.
But Its strange immediately after setting serial that the avail memory is 50k bytes less than what it was in the basic script, could this be my issue or am I making things up?
Basic script:
#include "I2S.h"
#define FREEMEM_CELL 10
struct elem {
struct elem *next;
char dummy[FREEMEM_CELL-2];
};
int getFreeMemory(void) {
int counter;
struct elem *head, *current, *nextone;
current = head = (struct elem*) malloc(sizeof(struct elem));
if (head == NULL) {
return 0;
}
counter = 0;
do {
counter++;
current->next = (struct elem*) malloc(sizeof(struct elem));
current = current->next;
} while (current != NULL);
current = head;
do {
nextone = current->next;
free(current);
current = nextone;
} while (nextone != NULL);
return counter * FREEMEM_CELL;
}
void setup() {
Serial.begin(115200);
while (!Serial) {} // Wait for serial connection.
// Print free memory before I2S initialization
Serial.print("Free memory before I2S init: ");
Serial.println(getFreeMemory());
// Initialize I2S
if (!I2S.begin(44100)) {
Serial.println("Failed to initialize I2S!");
} else {
Serial.print("Free memory after I2S init: ");
Serial.println(getFreeMemory());
}
}
void loop() {
// Print free memory periodically
Serial.print("> free memory: ");
Serial.println(getFreeMemory());
delay(1000); // Delay for 1000 milliseconds (1 second)
}
Voice model
//#include <Arduino.h>
#include <DSpotterSDK_MakerHL.h>
#include <LED_Control.h>
#include <I2S.h>
#include <SD.h>
#include <SPI.h>
// The DSpotter License Data.
#include "CybLicense.h"
#define DSPOTTER_LICENSE g_lpdwLicense
// The DSpotter Keyword Model Data.
#if defined(TARGET_ARDUINO_NANO33BLE) || defined(TARGET_PORTENTA_H7) || defined(TARGET_NICLA_VISION)
// For ARDUINO_NANO33BLE and PORTENTA_H7
#include "Model_L1.h" // The packed level one model file.
// For NANO_RP2040_CONNECT
#elif defined(TARGET_NANO_RP2040_CONNECT)
#include "Model_L0.h" // The packed level zero model file.
#endif
#define DSPOTTER_MODEL g_lpdwModel
#define FREEMEM_CELL 10
// The VR engine object. Only can exist one, otherwise not worked.
static DSpotterSDKHL g_oDSpotterSDKHL;
const int chipSelect = 29; // SD card chip select pin
File wavFile;
struct elem {
struct elem *next;
char dummy[FREEMEM_CELL-2];
};
int getFreeMemory(void) {
int counter;
struct elem *head, *current, *nextone;
current = head = (struct elem*) malloc(sizeof(struct elem));
if (head == NULL) {
return 0;
}
counter = 0;
do {
counter++;
current->next = (struct elem*) malloc(sizeof(struct elem));
current = current->next;
} while (current != NULL);
current = head;
do {
nextone = current->next;
free(current);
current = nextone;
} while (nextone != NULL);
return counter * FREEMEM_CELL;
}
// Callback function for VR engine
void VRCallback(int nFlag, int nID, int nScore, int nSG, int nEnergy)
{
if (nFlag==DSpotterSDKHL::InitSuccess)
{
//ToDo
}
else if (nFlag==DSpotterSDKHL::GetResult)
{
/*
When getting an recognition result,
the following index and scores are also return to the VRCallback function:
nID The result command id
nScore nScore is used to evaluate how good or bad the result is.
The higher the score, the more similar the voice and the result command are.
nSG nSG is the gap between the voice and non-command (Silence/Garbage) models.
The higher the score, the less similar the voice and non-command (Silence/Garbage) models are.
nEnergy nEnergy is the voice energy level.
The higher the score, the louder the voice.
*/
//ToDo
if (nID == 10000) {
// Read and send samples to I2S
while (wavFile.available()) {
int16_t sample = wavFile.read(); // Read low byte
sample |= wavFile.read() << 8; // Read high byte
I2S.write(sample); // Send sample to I2S (left channel)
I2S.write(sample); // Send same sample to I2S (right channel)
}
// Close file when done
wavFile.close();
while (1);
}
}
else if (nFlag==DSpotterSDKHL::ChangeStage)
{
switch(nID)
{
case DSpotterSDKHL::TriggerStage:
LED_RGB_Off();
LED_BUILTIN_Off();
break;
case DSpotterSDKHL::CommandStage:
LED_BUILTIN_On();
break;
default:
break;
}
}
else if (nFlag==DSpotterSDKHL::GetError)
{
if (nID == DSpotterSDKHL::LicenseFailed)
{
//Serial.print("DSpotter license failed! The serial number of your device is ");
//Serial.println(DSpotterSDKHL::GetSerialNumber());
}
g_oDSpotterSDKHL.Release();
while(1);//hang loop
}
else if (nFlag == DSpotterSDKHL::LostRecordFrame)
{
//ToDo
}
}
void setup()
{
// Init Serial output for debug info
Serial.begin(115200);
delay(1000);
// Print free memory before Cyberon model
Serial.print("Free memory start of init: ");
Serial.println(getFreeMemory());
// Print free memory before I2S initialization
Serial.print("Free memory before I2S init: ");
Serial.println(getFreeMemory());
// // Start I2S with a sample rate of 22500 Hz (change to 44100 Hz if needed)
// if (!I2S.begin(22500)) {
// Serial.println("Failed to initialize I2S!");
// } else{
// Serial.print("Free memory after I2S init: ");
// Serial.println(getFreeMemory());
// }
//while (!Serial); // Wait for Serial to be ready
DSpotterSDKHL::ShowDebugInfo(true);
// Init LED control
LED_Init_All();
// Initialize SD card
if (!SD.begin(chipSelect)) {
Serial.println("Failed to initialize SD card!");
}
// Open the WAV file
wavFile = SD.open("lords.wav");
if (!wavFile) {
Serial.println("Failed to open WAV file!");
}
// Skip WAV header (44 bytes for standard WAV files)
wavFile.seek(44);
// Print free memory before Cyberon model
Serial.print("Free memory before Cyberon model init: ");
Serial.println(getFreeMemory());
// Init VR engine & Audio
if (g_oDSpotterSDKHL.Init(DSPOTTER_LICENSE, sizeof(DSPOTTER_LICENSE), DSPOTTER_MODEL, VRCallback) != DSpotterSDKHL::Success) {
Serial.println("Failed to initialize DSpotter SDK!");
}
}
void loop()
{
// Do VR
g_oDSpotterSDKHL.DoVR();
}
The Nano RP2040 is supposed to have 264 kB of RAM. To the following minimal Arduino program add Serial and the print free memory code, and try to understand what the output is telling you.
Hey, what I was trying to say is that I tried a script that just ran I2S.begin. This had 80,000 bytes of available memory. When I put into voice recognition model it had 30,000bytes. These were both after initialising the serial.begin in the start up. So nothing had happened before.
After a search, it would seem that I would need roughly 4kb - 8kb memory space to run the I2S. Therefore I don't think memory is causing the hang. And will now look into PIN out's in libraries. Like suggested
I could not check inside DSpotterSDK_MakerHL.h as it seems private. I commented everything out of the code apart from the #include "I2S.h" and the dspottersdk. This is when the I2S initiation causes a hang.
So It's a conflict between I2S and the cyberon model. I know that the RP2040 does not have a I2S so it uses the PIO. My next thought is there could be a PIO (I2S) and a PCM conflict?
A glance at DSpotterSDKMakerHL.cpp clearly indicates that the library uses the PDM microphone.
/**
Callback function to process the data from the PDM microphone.
NOTE: This callback is executed as part of an ISR.
Therefore using `Serial` to print messages inside this function isn't supported.
**/
Thanks for sharing that and I am now bettered educated for it.