Using python based audio library in app lab

[n1lzer0]

Hello

For a project, i am intending to use Whisper STT in UNO Q. However it requires ffmpeg, onnx-runtime, rust and pytorch and i am not sure if it can be accesed in my python code within the app lab.

For the project, i wanted to record the audio via microphone, save it as mp3 file and then process it via the STT, so i wanted to know if i can access all type of modules from app lab as in a traditional IDE (for example pycharm) on a desktop.

Thanks

[ptillisch]

Hi @n1lzer0. I know it has been a long time since you posted this, but I wanted to respond since new capabilities have been recently been added to the Arduino App framework, which should facilitate this sort of use case. This may be useful to you, or to others who want to accomplish something similar.

You already identified this as a possible solution for another use case:

@n1lzer0

With the addition of custom bricks (the docker one) , is it possible now to somehow add the required packages using it?

I responded on that other topic to share some resources on the subject:

Unable to install llama cpp python library in App Lab

Hi @n1lzer0. You are definitely on the right track! The custom Bricks feature does not allow you to customize the contents of the primary Docker container in which the App's Python script runs. However, it does allow you to run additional custom containers, which the Python script can interface with. This is actually how some of the official Bricks provided by Arduino work already. Unfortunately we haven't published documentation for the custom Bricks feature yet, but the documentation team ha…

Note that the Docker container of custom Bricks may provide any type of resource. This is not limited to utilizing Python packages that have dependencies on resources not available in the App's primary container.

You can use a custom container to provide your Brick with access to things like the Rust programming language/toolchain, or Rust-based tools. You can also provide your container with access to hardware interfaces of the UNO Q such as a microphone. In turn, that Brick can make these capabilities available for use by your App.