[solved]FFT Beat Detection esp32

dear community,

I’m working on a project that is a LED matrix that reacts to the beat or rhythm of the music that’s playing. and I know it’s very hard to do.
like this one: https://www.instagram.com/p/B_5ThdBhUNb/

if possible I want to do the audio processing and analysing on an esp32, I want to use an Arduino to send out the data to the LEDs. But I have no experience with programming processing and analysing of audio.
The animations don’t need to be too hard to make or draw for the Arduino.

so I thought that it was a cool project to learn this “skill”. and went up being a rabbit hole of endless searching.

I have ordered and received 4 different microphones:
max9814, max4466, SparkFun Sound Detector and an electret microphone with an lm358

what I want it to do or can do:
FFT with beat detection, beat timing and silence detection

I already have some ideas on how it can be done.
but don’t know how I can code it into a functional program.

idea 1:
use a low pass filter in combination with an opamp
the low pass filter to filter out the higher frequencies and to the opamp to amplify the signal and feed it into the esp32 for filtering, processing and analysing of the signal

idea 2:

still use a low pass filter in combination with an opamp but feed it into an optocoupler.
the low pass filter to filter out the higher frequencies and to the opamp to amplify the signal and finally, feed the amplified signal in the optocoupler so the esp32 can detect the signal strength and the frequency.

for the processing and analysing of audio:

or

diod-dev uses a teensy in combination with an esp32 or an FFT library and do the coding by myself but I don’t want to reinvent the wheel.

I hope that someone could help me with this journey and finally get to a working program that works

when i worked on speakerphones which needed to detect speech, we used leaky integration with different attach and decay rates to measure speech and background noise.

avg += (sig - avg) * K;

where K is < 1 and depends if sig > avg.

when the attack rate > decay rate, avg captures a peak, while for the opposite is capture the noise level.

when I worked on heartbeat detection for my master's these, i i detected the peak in the signal by by looking for a sample that is >= than the previous and > than the following sample.

these techniques may be useful for detecting candidate samples for beat detection

I don't know if it can be used in my project. because it will be standing in a noisy environment, and want "the best" result. but still, it could be used in combination with the optocoupler.

also, I added a photo of the optocoupler to the first post.

with the optocoupler it's possible to detect the peaks and the amplitude.

i should have said that sig is the absolute value of the audio sample, it's energy, which effectively removes all non-DC components (i.e. filter the noise)

ohh, now I get it! So if I filter the noise (everything above 200 Hz) I will get a "proper" signal. and look at the peaks in that signal. (testing it with your bit of code).

I will give it a try! today or tomorrow I will post an update. one more question for now. what's K used for? To amplify the signal or something else?

avg += (sig - avg) * K;

K is the averaging coefficient. I think of it an an RC time constant. If sig is a step function avg become close to the value of sig after 3/K iterations.

i think i'm wrong in saying that taking the absolute value filters out the AC, but aren't you just looking or an energy measure?

how much noise is there and what is the spectra of the noise? if the beat you're looking for at such a low level that higher frequency noise makes it hard to hear?

doesn't a pulse require infinite frequency response?

sebasdt1: ohh, now I get it! So if I filter the noise (everything above 200 Hz) I will get a "proper" signal. and look at the peaks in that signal. (testing it with your bit of code).

I will give it a try! today or tomorrow I will post an update. one more question for now. what's K used for? To amplify the signal or something else?

avg += (sig - avg) * K;

The operation of this formula might be more clear if expanded to the equivalent:

avg = avg * (1-K) + sig * K

If, for example K=1/8, then the new average is 7/8 parts of the running average and 1/8 part of the most recent sample.

A commonly used efficient implementation constarains K to be the reciprocal of a power of 2, e.g. K = 1/2N where N is an integer. With this the "leaky integrator" can implemented using shift operations rather than multiplies which is much faster on a processor that does not support hardware multiply like the 8-bit Arduinos:

avg = avg - avg >> N + sig >> N

Noting that the shift loses N of the least significant bits of "sig", a better implementation maintains the running average left shifted N bits and only right shifts when returning the average:

avgLeftShiftN = avgLeftShiftN - avg + sig
avg = avgLeftShiftN >> N

Applying this filter to the absolute value of the signal as "gcjr" has proposed is a form of an envelop follower, commonly used for AM detection. For beat detection the idea is that the amplitude peaks on the beat are higher than the threshold set by "avg" (or a scaled version thereof) which is generally the case for music with a prominent beat (e.g. drum hits on a dance track) and typically not the case for music with sustained notes and even amplitude (e.g. a string quartet).

I did the project to show lights blinking to the music, works great. I did edit the AudioAnalyzer cpp to use the ESP32 A:D API, which produces better results then using the default analog to digitial settings.

#include "sdkconfig.h"
#include "driver/rtc_io.h"
#include "esp_system.h" //This inclusion configures the peripherals in the ESP system.
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/timers.h"
#include "freertos/event_groups.h"
#include 
#include "AudioAnalyzer.h"
////
/* define event group and event bits */
EventGroupHandle_t eg;
#define evtDo_AudioReadFreq       ( 1 << 0 ) // 1
////
QueueHandle_t xQ_LED_Info;
////
const int LED_COUNT = 24; //total number of leds in the strip
////
// When we setup the NeoPixel library, we tell it how many pixels, and which pin to use to send signals.
Adafruit_NeoPixel leds = Adafruit_NeoPixel( LED_COUNT, 26, NEO_GRB + NEO_KHZ800 );
////
void setup()
{
  eg = xEventGroupCreate();
  //  Audio.Init(); // start the audio analyzer
  //  leds.begin(); // Call this to start up the LED strip.
  //  clearLEDs( LED_COUNT );  // This function, defined below, de-energizes all LEDs...
  //  leds.show();  // ...but the LEDs don't actually update until you call this.
  ////
  int FreqVal[7];
  xQ_LED_Info = xQueueCreate ( 1, sizeof(FreqVal) );
  //////////////////////////////////////////////////////////////////////////////////////////////
  xTaskCreatePinnedToCore( fDo_AudioReadFreq, "fDo_ AudioReadFreq", 40000, NULL, 4, NULL, 1 ); //assigned to core
  xTaskCreatePinnedToCore( fDo_LEDs, "fDo_ LEDs", 40000, NULL, 4, NULL, 0 ); //assigned to core
  xEventGroupSetBits( eg, evtDo_AudioReadFreq );
} // setup()
////
void loop() {} // void loop
////
void fDo_LEDs( void *pvParameters )
{
  const int Brightness = 180;
  const int SEG = 6; // how many parts you want to separate the led strip into
  const int ledCount = LED_COUNT; //total number of leds in the strip
  int iFreqVal[7];
  int j;
  leds.begin(); // Call this to start up the LED strip.
  clearLEDs( ledCount );  // This function, defined below, de-energizes all LEDs...
  leds.show();  // ...but the LEDs don't actually update until you call this.
  leds.setBrightness( Brightness ); //  1 = min brightness (off), 255 = max brightness.
  for (;;)
  {
    if (xQueueReceive( xQ_LED_Info, &iFreqVal,  portMAX_DELAY) == pdTRUE)
    {
      j = 0;
      //assign different values for different parts of the led strip
      for (j = 0; j < ledCount; j++)
      {
        if ( (0 <= j) && (j < (ledCount / SEG)) )
        {
          set( j, iFreqVal[0], ledCount, SEG ); // set the color of led
        }
        else if ( ((ledCount / SEG) <= j) && (j < (ledCount / SEG * 2)) )
        {
          set( j, iFreqVal[0], ledCount, SEG );
        }
        else if ( ((ledCount / SEG * 2) <= j) && (j < (ledCount / SEG * 3)) )
        {
          set( j, iFreqVal[0], ledCount, SEG );
        }
        else if ( ((ledCount / SEG * 3) <= j) && (j < (ledCount / SEG * 4)) )
        {
          set( j, iFreqVal[0], ledCount, SEG );
        }
        else if ( ((ledCount / SEG * 4) <= j) && (j < (ledCount / SEG * 5)) )
        {
          set( j, iFreqVal[0], ledCount, SEG );
        }
        else
        {
          set( j, iFreqVal[0], ledCount, SEG );
        }
      }
      leds.show();
    }
    xEventGroupSetBits( eg, evtDo_AudioReadFreq );
  }
  vTaskDelete( NULL );
} // void fDo_ LEDs( void *pvParameters )
////
void fDo_AudioReadFreq( void *pvParameters )
{
  int FreqVal[7];
  const int NOISE = 10; // noise that you want to chop off
  const int A_D_ConversionBits = 4096; // arduino use 1024, ESP32 use 4096
  Analyzer Audio = Analyzer( 5, 15, 36 );//Strobe pin ->15  RST pin ->2 Analog Pin ->36
  Audio.Init(); // start the audio analyzer
  int64_t EndTime = esp_timer_get_time();
  int64_t StartTime = esp_timer_get_time(); //gets time in uSeconds like Arduino Micros
  for (;;)
  {
    xEventGroupWaitBits (eg, evtDo_AudioReadFreq, pdTRUE, pdTRUE, portMAX_DELAY);
    EndTime = esp_timer_get_time() - StartTime;
    // log_i( "TimeSpentOnTasks: %d", EndTime );
    Audio.ReadFreq(FreqVal);
    for (int i = 0; i < 7; i++)
    {
      FreqVal[i] = constrain( FreqVal[i], NOISE, A_D_ConversionBits );
      FreqVal[i] = map( FreqVal[i], NOISE, A_D_ConversionBits, 0, 255 );
      // log_i( "Freq %d Value: %d", i, FreqVal[i]);//used for debugging and Freq choosing
    }
    xQueueSend( xQ_LED_Info, ( void * ) &FreqVal, 0 );
    StartTime = esp_timer_get_time();
  }
  vTaskDelete( NULL );
} // fDo_ AudioReadFreq( void *pvParameters )
////
//the following function set the led color based on its position and freq value
//
void set(byte position, int value, int ledCount, int segment)
{
  // segment 0, red
  if ( (0 <= position) && (position < ledCount / segment) ) // segment 0 (bottom to top), red
  {
    if ( value == 0 )
    {
      leds.setPixelColor( position, 0, 0, 0 );
    }
    else
    {
      // increase light output of a low number
      // value += 10;
      // value = constrain( value, 0, 255 ); // keep raised value within limits
      leds.setPixelColor( position, leds.Color( value , 0, 0) );
    }
  }
  else if ( (ledCount / segment <= position) && (position < ledCount / segment * 2) ) // segment 1 yellow
  {
    if ( value == 0 )
    {
      leds.setPixelColor(position, leds.Color(0, 0, 0));
    }
    else
    {
      leds.setPixelColor(position, leds.Color( value, value, 0)); // works better to make yellow
    }
  }
  else if ( (ledCount / segment * 2 <= position) && (position < ledCount / segment * 3) ) // segment 2 pink
  {
    if ( value == 0 )
    {
      leds.setPixelColor(position, leds.Color(0, 0, 0));
    }
    else
    {
      leds.setPixelColor(position, leds.Color( value, 0, value * .91) ); // pink
    }
  }
  else if ( (ledCount / segment * 3 <= position) && (position < ledCount / segment * 4) ) // seg 3, green
  {
    if ( value == 0 )
    {
      leds.setPixelColor(position, leds.Color( 0, 0, 0));
    }
    else //
    {
      leds.setPixelColor( position, leds.Color( 0, value, 0) ); //
    }
  }
  else if ( (ledCount / segment * 4 <= position) && (position < ledCount / segment * 5) ) // segment 4, leds.color( R, G, B ), blue
  {
    if ( value == 0 )
    {
      leds.setPixelColor(position, leds.Color( 0, 0, 0));
    }
    else //
    {
      leds.setPixelColor(position, leds.Color( 0, 0, value) ); // blue
    }
  }
  else // segment 5
  {
    if ( value == 0 )
    {
      leds.setPixelColor(position, leds.Color( 0, 0, 0)); // only helps a little bit in turning the leds off
    }
    else
    {
      leds.setPixelColor( position, leds.Color( value, value * .3, 0) ); // orange
    }
  }
} // void set(byte position, int value)
////
void clearLEDs( int ledCount)
{
  for (int i = 0; i < ledCount; i++)
  {
    leds.setPixelColor(i, 0);
  }
} // void clearLEDs()

Reading post #7 and quickly looking at the code, my first thought was, "Ok, that's a spectral display, it doesn't really do 'beat detection'" at least as I'd interpreted the original post.

On second thought, if one's goal is really a light display that is somehow correlated with the music, does one really need to do beat detection in the sense that the program specifically identifies specific points in time where the beat occurs or is it sufficient that the light display is synchronized to the music leaving the observer to see the beat that emerges? If the objective is beat detection, say lighting a single LED on the beat, then some detection algorithm is required. If the objective is a interesting light display, then lighting some number of LEDs in proportion to the amplitude from an FFT frequency bin will display a beat pattern for music with a prominent amplitude beat.

sebasdt1: like this one: https://www.instagram.com/p/B_5ThdBhUNb/

MrMark: Reading post #7 and quickly looking at the code, my first thought was, "Ok, that's a spectral display, it doesn't do 'beat detection'" at least as I'd interpreted the original post.

That's why I gave a link to an example, my goal is to reach that kind of detection. And I wasn't intending to do a spectral display. and I also know that an algorithm is needed for this of detection.

MrMark: On second thought, if one's goal is a light display that is somehow correlated with the music, does one need to do beat detection in the sense that the program specifically identifies specific points in time where the beat occurs or is it sufficient that the light display is synchronized to the music leaving the observer to see the beat that emerges? If the objective is an interesting light display, then lighting some number of LEDs in proportion to the amplitude from an FFT frequency bin will display a beat pattern for music with a prominent amplitude beat.

that's exactly what I want it to do at some point!

Idahowalker: I did the project to show lights blinking to the music, works great. I did edit the AudioAnalyzer cpp to use the ESP32 A:D API, which produces better results then using the default analog to digitial settings.

Do you know what you changed in the CPP file? but thanks for sharing this bit of code.

AudioAnalyizer.cpp

/*
AudioAnalyzer.cpp - Library for audio spectrum analyzer.
Created by Lauren Pan,Dec 5, 2012.
E-Mail: Lauren.pan@dfrobot.com

Version 1.3

Log:
2012.12.05
Improve the code to make it compatible with the Arduino IDE 1.0 or later

2010.11.06
Add optional analog pin
*/
#include 
#include "AudioAnalyzer.h"
#if defined(ARDUINO) && ARDUINO >= 100
#include "Arduino.h"
#else
#include "WProgram.h"
#endif

/**************************** Init Analyzer connecter Pin ****************************/
Analyzer::Analyzer(void)
{
    _StrobePin = 4;
    _RSTPin = 5;
    _DCPin = 0;
}

Analyzer::Analyzer(int StrobePin,int RstPin,int AnalogPin)
{
    _StrobePin = StrobePin;
    _RSTPin = RstPin;
    _DCPin = AnalogPin;
 // set up A:D channels
  adc1_config_width(ADC_WIDTH_12Bit);
  adc1_config_channel_atten(ADC1_CHANNEL_0, ADC_ATTEN_DB_11); // esp32 GPIO36
}

void Analyzer::Init()
{
    pinMode(_StrobePin,OUTPUT);
    pinMode(_RSTPin,OUTPUT);
    RstModule();
}
/**************************** Reset analyzer module ****************************/
void Analyzer::RstModule()
{
    digitalWrite(_StrobePin,LOW);
    digitalWrite(_RSTPin,HIGH);
    digitalWrite(_StrobePin,HIGH);
    digitalWrite(_StrobePin,LOW);
    digitalWrite(_RSTPin,LOW);
    delayMicroseconds(72);  
}

/**************************** Read DC out value ****************************/
void Analyzer::ReadFreq(int *value)
{
    static boolean RstState = false;
    if(!RstState)
    {
        _TimepointSt = millis();
        RstState = true;
    }
    else
    {
        _TimepointNow = millis();
        if(_TimepointNow - _TimepointSt > 3000)
        {
            RstModule();
            RstState = false;
            //Serial.println("Rst");
        }
    }

    for(byte band = 0;band

do you have a link to the library? I couldn't find it in the ide library.

try:

internet, search , words: "dfr audioanalyizer.cpp"

are you trying to build what use to be called a "color organ"?

some hardware bandwidth filters could be used to cover the range from 100-20,000 Hz and their power output used to modulate the intensity of light/leds

of course an FFT could be used and ranges of bin added

presumably a beat of the music would be visual

I uploaded a video to youtube so you can see what I’m working on. in this video it isn’t finished yet.
on this led matrix I’m planning to program 2 modes:
demo and audio.
for now, I want to focus on the audio mode.

the esp32 processes the incoming audio and looks for a beat and if there is a beat. then display something, just like in that Instagram post.

Idahowalker: try:

internet, search , words: "dfr audioanalyizer.cpp"

I tried but only found this github link. https://github.com/talentspsp/audio_analysis\ downloaded it but it didn't want to work. the compiler gave an error

It's customary here to cut and paste an error to a message post, instead of asking people to imagine what it might be.

sebasdt1:
I tried but only found this github link. GitHub - talentspsp/audio_analysis
downloaded it but it didn’t want to work. the compiler gave an error

tisk, tisk Audio_Analyzer__SKU_DFR0126_-DFRobot

aarg: It's customary here to cut and paste an error to a message post, instead of asking people to imagine what it might be.

Sorry I thought that I posted the error code.

AudioAnalyzer.h: No such file or directory.

But I put the library in the folder and the library isn't in the example menu.

I have searched for a problem but couldn't find an obvious error. then the compiler gave an error: invalid library found in C:...\Documents\Arduino\libraries\AudioAnalyzer_v1.2: no headers files (.h) found in C:...\Documents\Arduino\libraries\AudioAnalyzer_v1.2

Then I moved the .ccp and .h file from: Documents\Arduino\libraries\AudioAnalyzer_v1.2\AudioAnalyzer to: Documents\Arduino\libraries\AudioAnalyzer_v1.2\ and that folder was now empty:

then the compiler gave another error: Documents\Arduino\libraries\AudioAnalyzer_v1.2/AudioAnalyzer.h:10:22: fatal error: WProgram.h: No such file or directory and indeed there was no file there. I don't know what this .h file does.