Arduino Forum

Products => MKR Boards => MKRVIDOR4000 => Topic started by: loopmasta on Jul 25, 2018, 05:49 pm

Title: VidorDSP
Post by: loopmasta on Jul 25, 2018, 05:49 pm
- I would love to use a VidorDSP lib to build a synth out of basic dsp blocks with input / output via USB audio and controlled / patched via Arduino code.

- It would be great if the VidorDSP could read / write from / to RAM so we could implement samplers or use RAM as a audio buffer to pipe processed audio from one dsp block to another.

- A sample accurate event driven API would be useful to write sequencers.

- Oscillators / WaveTables / SamplePlayer / Filters / LFOs / Envelopers / Mixers / FFT / Effects / TimerEvents / ...


I am getting exited. Just some random thoughts.
Title: Re: VidorDSP
Post by: DarioPennisi on Jul 26, 2018, 09:25 am
Hi Loopmasta,
thank you for your input. let's use this thread to define how it would look like...

1) we can input and output audio in FPGA through I2S, either passing through SAM D21 acting as USB audio device or via an external CODEC (we're thinking about a shield for this). we can also output audio via any FPGA pin as we have a sigma delta DAC IP running at 100 MHz where you can modulate sampling frequency/dynamic range at your will

2) we have SDRAM which can be used for buffering either pre-sampled audio clips or to perform delays. SDRAM functionality is not optimal for random access but with proper caching/buffering in internal RAM this is viable. memory is clockable up to 140 MHz but in reality we should likely keep that down to 100 MHz but that is still quite a lot of data, and even considering overhead we could read/write up to around 80 MSPS.

3) for oscillators the easiest thing is creating a wave lookup table using embedded RAM. each embedded RAM block can hold 512 samples at 16 bit and we already have a "reader" that can run at fractional frequency so that you can play that back at any frequency you like. this can generate any waveform, provided the 512 samples are enough. we could also  use SDRAM for this but that would make it a bit more complex as we need to have buffering to compensate for latency

4) most logic inside the FPGA can run at high frequency so a single IP can process multiple channels within a single sample timing. for example if i have a single multiplier running at 100 MHz and my sampling frequency is 50 KHz i can use this multiplier 2000 times per sample so for example i can have a 2K tap FIR filter with a single multiplier or i can mix 2K channels each with individual volumes

5) in order to control the whole system we can have a soft processor core running in the FPGA program routing of audio samples across IP blocks. at audio sampling frequencies this should be relatively trivial (a NIOS processor runs at 100 MHZ easily). SAM D21 can in parallel handle other stuff such as USB communication, interface with sensors/actuators, etc.


thinking about the way it could work, the core of this system may be a SDRAM scheduler which basically feeds each IP block with packets of data and stores output packets of data. each packet would be a "channel and by defining which channel goes to which IP in which sequence would define routing. by just swapping buffer addresses. actually passing through SDRAM means we should do bursts so "sample accurate" may have to be approximated to the burst size... let me know if this would be acceptable..
let's say then that with 80 MSPS, at 40 KHz we have 2K channels. these have to be divided by two because we have to read and write these channels so it's 1K channels which means we may have up to 1K "virtual" processing blocks running in parallel. it's rough estimates but gives you an idea of what can be done...

now, fire is started... let's keep discussing...
Title: Re: VidorDSP
Post by: loopmasta on Jul 27, 2018, 12:22 am
Hi Dario, thanks for replying to my random notes. Thats a lot of information in one post but it all sounds more than promising. Its now a little too late (round midnight) for a constructive reply but i'll be back tomorrow.


Title: Re: VidorDSP
Post by: loopmasta on Jul 28, 2018, 02:06 pm
The hardware specs of the board are impressive but I would like to take a step back and create a user story first to see if we are talking about a similar user case.

I've tried a lot of Arduino audio libs but the one that i liked the most is the Teensy Audio Library with its patcher. So i as a VidorAudio user i would love to write code like this to create a simple synth for example.

!!! FAKE PSEUDOCODE BELOW !!!

#include "VidorAudio.h"

Vidor_SYNTH synth;
Vidor_PATCHER patch;
Vidor_USB out;

Vidor_OSC osc1;
Vidor_OSC osc2;
Vidor_NOISE noise;
Vidor_MIX mix;
Vidor_SVF svf;
Vidor_ENV env1;
Vidor_ENV env2;
Vidor_AMP amp;

// Setup VidorAudio
void setup() {
  Serial.begin(115200);
  while (!Serial){}

  // Initialize the FPGA
  if (!FPGA.begin()) {
    Serial.println("Initialization failed!");
    while (1) {}
  }

  delay(4000);
  Serial.println("Power ON");

  // Patch the synth & blocks
  mix.init(3);
  patch.init(8 );
  synth.init(patch, out);

  patch.set(1, osc1.out(1), mix.in(1));
  patch.set(2, osc2.out(1), mix.in(2));
  patch.set(3, noise.out(1), mix.in(3));

  patch.set(4, mix.out(1), svf.in(1));
  patch.set(5, svf.out(1), amp.in(1));
  patch.set(6, amp.out(1), out.in(1));

  patch.set(7, env1.out(1), swf.in(2));
  patch.set(8, env2.out(1), amp.in(2));
}

// Main loop
void loop() {
  // listen to midi events
  if (midi) {
      // Do midi stuff
      if (midi.on) {
        synth.play(frequency, velocity);
      } else if (midi.off) {
        synth.stop(frequency, velocity);
      } else {
        synth.change(id, value);
      }
  }

  // Listen to pot / button / switch changes
  if (controls) {
      // Do hardware stuff
      osc1.set(mode, pitch, glide);
      osc2.set(mode, pitch, glide);
      noise.set(mode);
      svf1.set(mode, freq, q, envAmount);
      mix1.set(level1, level2, level3);
      env1.set(attack, decay, sustain, release);
      env2.set(attack, decay, sustain, release);
      amp1.set(volume);
  }
}

The first question is. Is something like this possible or do we need bigger blocks with a less patchable structure. If it is possible we should first define what a block is and how it exposes its type and connections.
Title: Re: VidorDSP
Post by: DarioPennisi on Jul 28, 2018, 03:02 pm
Yes, we're on the same page even if likely you won't declare blocks as standalone objects but rather find them within a dsp class that would get automatically populated with resources compiled in the fpga...
For the rest I'd say sample processing would be done entirely in fpga and controls mostly on Sam with offloading by the internal soft core.
Title: Re: VidorDSP
Post by: loopmasta on Jul 30, 2018, 09:43 pm
Ok. But how do you patch blocks if you don't have a reference pointer? Or how do you create three instances of the same block? I must admit that i'am having a hard time to embed the concept of FPGA's into my thinking workflow.

Title: Re: VidorDSP
Post by: DarioPennisi on Jul 30, 2018, 10:30 pm
my idea is that there would be a DSP object which would automatically allocate the "sub objects" based on what is contained in the FPGA. this means that at creation time it would query the FPGA which will return the list of available hardware resources and then create an object for every resource. from the user perspective you would have a query that would return the list of available resources and a getResource api that would return you the next instance of a given resource, if available.
with those objects you could then call the patch API that in turn would inform the FPGA on how to route it internaly.
now.. given that some HW resources may be time multiplexed they woud result in multiple virtual resources. for this reason maybe it would not make much sense to create a real object for each resource but rather a class that you would use to reference an handler which in turn is just an index for the hw resource and a subindex for its multiplex index.
maybe got too technical but it's just to share some ideas on how the hw/sw can be optimized...
Title: Re: VidorDSP
Post by: loopmasta on Jul 31, 2018, 01:33 pm
my idea is that there would be a DSP object which would automatically allocate the "sub objects" based on what is contained in the FPGA. this means that at creation time it would query the FPGA which will return the list of available hardware resources and then create an object for every resource. from the user perspective you would have a query that would return the list of available resources and a getResource api that would return you the next instance of a given resource, if available.
with those objects you could then call the patch API that in turn would inform the FPGA on how to route it internaly.
Ok. That sound great.

now.. given that some HW resources may be time multiplexed they woud result in multiple virtual resources. for this reason maybe it would not make much sense to create a real object for each resource but rather a class that you would use to reference an handler which in turn is just an index for the hw resource and a subindex for its multiplex index.
Ok. I think i have to see this in action to fully understand it but i'll leave that for later.

maybe got too technical but it's just to share some ideas on how the hw/sw can be optimized...
No its fine. My goal is to learn more about the technical aspects of working with FPGAs.


A few more questions.

+ Is it possible to access the HDMI port while running a VideoAudio project? It would be great to have a simple oscilloscope block to display the audio output.

+ Is it possible to create multiple instances of a patch to create a polyphonic instrument?
Title: Re: VidorDSP
Post by: DarioPennisi on Jul 31, 2018, 03:40 pm
Hi,
yes, it is possible to have video running along with Audio but since memory bandwidth is limited that would impact the amount of patches or time multiplexing you can do.

as long as there are enough resources either in number of HW blocks or in terms of time multiplexing you can do as many patches as you want. for example, let's say you have a multiplier, an accumulator and a coefficient memory. with these you can do a FIR filter. if you can time multiplex them, say 1000 times per sample then you can have a 1000 tap filter or 2x500 filters so on.

of course since we do time multiplexing each patch would have an iteration number so that patches that need to process just one sample would be active one time per sample while patches such as the ones for the FIR would be active multiple times per sample.
Title: Re: VidorDSP
Post by: loopmasta on Aug 02, 2018, 12:11 am
I am so used to hardware limitation when doing audio dsp stuff in software that this sounds almost too good to be true. :-) Do we have to care about oversampling / bandlimited oscillators / aliasing at all? Do you have a timeline on when to expect a early version of a VidorAudio implementation?