Building a MIDI Piano with ESP32: Saving MIDI Output to Files

[nickr7]

Hello,

I'm a beginner in the field of microcontrollers and I'm working on a project to create a piano that generates MIDI notes for audio output. I've successfully implemented MIDI note generation, but now I want to extend the functionality to save the MIDI output to MIDI files for playback. It's important to maintain accurate timing, pitch, and note durations in the saved MIDI files.

For my setup, I'm using an ESP32 microcontroller along with a microSD card module for storage. The audio output is handled by a UDA1334A module connected to a JBL speaker, and for the button matrix, I'm using an SX1509.

The Code right now:

#include <Wire.h>
#include <SparkFunSX1509.h>
#include "AudioTools.h"
#include "StkAll.h"

#define KEY_ROWS 8
#define KEY_COLS 8


const byte SX1509_ADDRESS = 0x3E;
SX1509 io;

I2SStream i2s;
ArdStreamOut output(i2s, 1);
BlowBotl  blowbotl  ;
Voicer voicer;
int group = 1;
float amplitude = 30;
int note = 0;

const byte gedrueckt = LOW;
const byte losgelassen = HIGH;

// Array für die MIDI-Notenwerte
const byte midiNotes[KEY_ROWS][KEY_COLS] = {
  { 0, 52, 44, 36, 28, 20, 12, 0 },
  { 0, 53, 45, 37, 29, 21, 13, 0 },
  { 0, 54, 46, 38, 30, 22, 14, 0 },
  { 0, 55, 47, 39, 31, 23, 15, 0 },
  { 0, 56, 48, 40, 32, 24, 16, 0 },
  { 0, 57, 49, 41, 33, 25, 17, 0 },
  { 0, 58, 50, 42, 34, 26, 18, 0 },
  { 0, 59, 51, 43, 35, 27, 19, 0 }
};

void setup() {
  Serial.begin(115200);


  while (!Serial);
  Serial.println("Setup-Start");

  Wire.begin();
  Serial.println("Wire.begin() done");

  if (io.begin(SX1509_ADDRESS) == false) {
    Serial.println("Failed to communicate. Check wiring and address of SX1509.");
    while (1);
  }
  voicer.addInstrument(&blowbotl, group);

  auto cfg = i2s.defaultConfig(TX_MODE);
  cfg.bits_per_sample = 16;
  cfg.sample_rate = Stk::sampleRate();
  cfg.channels = 1;
  cfg.pin_bck = 27;   //#define I2S_BCLK  27
  cfg.pin_ws = 26;    //#define I2S_LRC   26
  cfg.pin_data = 25;  //#define I2S_DOUT  25
  i2s.begin(cfg);


  for (int c = 0; c < 8; c++) {
    io.pinMode(c, OUTPUT); //8er-Leiste ist OUTPUT
    Serial.print("io.pinMode(");
    Serial.print(c);
    Serial.println("), OUTPUT");
    // Serial.print("8er Leiste auf Output: ");
    Serial.println(io.digitalRead(c));
  }

  for (int r = 8; r < 16; r++) {
    io.pinMode(r, INPUT_PULLUP);
    Serial.println(io.digitalRead(r));
    Serial.print("io.pinMode(");
    Serial.print(r);
    Serial.println("), INPUT_PULLUP");
  }
}




void loop() {
  int col = 0;
  int row = 0;
  int rowx = 0;
  long endTime = millis() + 500;
  for ( row = 0; row < 8; row++) {
    io.digitalWrite(row, HIGH); 
  }

  for ( rowx = 0; rowx < 8; rowx++) {
    io.digitalWrite(rowx, LOW); 
    for ( col = 8; col < 16; col++) {
      while (io.digitalRead(col) == gedrueckt) {
        byte midiNote = midiNotes[rowx][col - 8];

        voicer.noteOn(midiNote, amplitude, group);
        while (millis() < endTime) {
          output.tick( voicer.tick() );
        }
        Serial.print("Note: ");
        Serial.println(midiNote);
      }

    }
    io.digitalWrite(rowx, HIGH); 
  }
}

That is my Harware right now (except the battery):

I'd appreciate any advice or experiences you may have with implementing MIDI file saving in this context.

Thank you in advance!

[Railroader]

@nickr7

For my setup

Is it a high secret?

Please read and use this topic: How to get the best out of this forum - Using Arduino / Project Guidance - Arduino Forum

[Deva_Rishi]

Initially i guess you should just store all events with a millis() timecode.
Any note on or note off, simplest solution would be to just store a command and timecode as a 7 byte struct.

You could afterwards convert this to a midi file format if you want, or just play it back as is.

I have not done this for Midi, but i have done this with much more data in the same way and it works just fine. In fact you can probably get away with just using SPIFFS if you'd want to.

[nickr7]

Thanks for your answer.

@Deva_Rishi

Initially i guess you should just store all events with a millis() timecode.
Any note on or note off, simplest solution would be to just store a command and timecode as a 7 byte struct.

It sounds logical, but could you explain it to me in more detail or provide an example?

I currently understand the function of Millis() as a timer that counts up from the start of the program. From there, you can use it to run something for a specific duration, or define a time value.

However, I don't have a clear idea of what such a time code might look like.

[Deva_Rishi]

OK well say you want to start recording, you assign the value of millis() to variable to reference the moment you start.

#include <MIDI.h>

uint32_t  recStart;    // millis() will return an unsigned 32 -bit integer or unsigned long

MIDI_CREATE_DEFAULT_INSTANCE();

void setup() {
}

void loop() {
  static bool isRecording = false;
  if (StartRecording()) {
    recStart = millis();
    isRecording = true;  
    // there is some more stuff to do here, initiate the file with name etc.
  }
  if (MIDI.read()) {
    midi::MidiType type = MIDI.getType();
    midi::DataByte d1 = MIDI.getData1();
    midi::DataByte d2 = MIDI.getData2();
    midi::Channel ch = MIDI.getChannel();
    if (isRecording) {
      uint32_t timeCode = millis() - recStart;
      // and then write to the file
      // - the timeCode as 32-bit integer, either lsb first or msb first as you like, but be consistent
      // and the type and both data bytes. some types don't have databytes but i would still write them to make the whole
      // command the same size. Most of the single byte commands probably shouldn't be written anyway.
      // the channel is optional, you may even choose to write the type + channel packed they way they come
    }
  }
}

bool StartRecording(){  // checks if recording should start
  return true;  // if it should, you have to determine the conditions
}

when playing back, again you assign the value of millis() and simply compare the next timeCode value in the file to see if the time to send that command has passed already. If so then you send it and wait for the next timeCode to have elapsed.

Hopefully this is clear enough.

There are more fancy programming techniques like creating a struct that contains those variables, but it is not required. All that is required is that when a command is received, you 'record' the time(millis() ) at that moment, substract the time when you started recording and store that result in the file. Then store the midi-command itself in the file.

If you want to convert that to an [actual midi file](MIDI - Wikipedia afterwards that can be done of course.

You could also go for the midi file format straight away, but i think what you need is more simple.

[nickr7]

I used this code for testing the microSD modul:

#include "FS.h"
#include "SD.h"
#include "SPI.h"

#define SCK   18  
#define MISO  19   
#define MOSI  23  
#define SD_CS 15 

SPIClass custom_spi = SPIClass(HSPI);
const unsigned long SPI_ClockHz = 80000;

void listDir(fs::FS &fs, const char * dirname, uint8_t levels) {
  Serial.printf("Listing directory: %s\n", dirname);

  File root = fs.open(dirname);
  if (!root) {
    Serial.println("Failed to open directory");
    return;
  }
  if (!root.isDirectory()) {
    Serial.println("Not a directory");
    return;
  }

  File file = root.openNextFile();
  while (file) {
    if (file.isDirectory()) {
      Serial.print("  DIR : ");
      Serial.println(file.name());
      if (levels) {
        listDir(fs, file.name(), levels - 1);
      }
    } else {
      Serial.print("  FILE: ");
      Serial.print(file.name());
      Serial.print("  SIZE: ");
      Serial.println(file.size());
    }
    file = root.openNextFile();
  }
}

void createDir(fs::FS &fs, const char * path) {
  Serial.printf("Creating Dir: %s\n", path);
  if (fs.mkdir(path)) {
    Serial.println("Dir created");
  } else {
    Serial.println("mkdir failed");
  }
}

void removeDir(fs::FS &fs, const char * path) {
  Serial.printf("Removing Dir: %s\n", path);
  if (fs.rmdir(path)) {
    Serial.println("Dir removed");
  } else {
    Serial.println("rmdir failed");
  }
}

void readFile(fs::FS &fs, const char * path) {
  Serial.printf("Reading file: %s\n", path);

  File file = fs.open(path);
  if (!file) {
    Serial.println("Failed to open file for reading");
    return;
  }

  Serial.print("Read from file: ");
  while (file.available()) {
    Serial.write(file.read());
  }
  file.close();
}

void writeFile(fs::FS &fs, const char * path, const char * message) {
  Serial.printf("Writing file: %s\n", path);

  File file = fs.open(path, FILE_WRITE);
  if (!file) {
    Serial.println("Failed to open file for writing");
    return;
  }
  if (file.print(message)) {
    Serial.println("File written");
  } else {
    Serial.println("Write failed");
  }
  file.close();
}

void appendFile(fs::FS &fs, const char * path, const char * message) {
  Serial.printf("Appending to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if (!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if (file.print(message)) {
    Serial.println("Message appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();
}

void renameFile(fs::FS &fs, const char * path1, const char * path2) {
  Serial.printf("Renaming file %s to %s\n", path1, path2);
  if (fs.rename(path1, path2)) {
    Serial.println("File renamed");
  } else {
    Serial.println("Rename failed");
  }
}

void deleteFile(fs::FS &fs, const char * path) {
  Serial.printf("Deleting file: %s\n", path);
  if (fs.remove(path)) {
    Serial.println("File deleted");
  } else {
    Serial.println("Delete failed");
  }
}

void testFileIO(fs::FS &fs, const char * path) {
  File file = fs.open(path);
  static uint8_t buf[512];
  size_t len = 0;
  uint32_t start = millis();
  uint32_t end = start;
  if (file) {
    len = file.size();
    size_t flen = len;
    start = millis();
    while (len) {
      size_t toRead = len;
      if (toRead > 512) {
        toRead = 512;
      }
      file.read(buf, toRead);
      len -= toRead;
    }
    end = millis() - start;
    Serial.printf("%u bytes read for %u ms\n", flen, end);
    file.close();
  } else {
    Serial.println("Failed to open file for reading");
  }


  file = fs.open(path, FILE_WRITE);
  if (!file) {
    Serial.println("Failed to open file for writing");
    return;
  }

  size_t i;
  start = millis();
  for (i = 0; i < 2048; i++) {
    file.write(buf, 512);
  }
  end = millis() - start;
  Serial.printf("%u bytes written for %u ms\n", 2048 * 512, end);
  file.close();
}

void setup() {
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }
  Serial.println("Setup-Start");
  custom_spi.begin(SCK, MISO, MOSI, SD_CS);
  Serial.print("custom_spi.begin(SCK=");
  Serial.print(SCK);
  Serial.print(", MISO=");
  Serial.print(MISO);
  Serial.print(", MOSI=");
  Serial.print(MOSI);
  Serial.print(", SD_CS=");
  Serial.print(SD_CS);
  Serial.println(" done");

  Serial.println("Mounting SD-Card");

  Serial.print("SD.begin(SD_CS_pin=");
  Serial.print(SD_CS);
  Serial.print("custom_spi SPI_ClockHz=");
  Serial.println(SPI_ClockHz);
  
  // chip-select-pin, SPI-instance, Bus-frequency
  if (!SD.begin(SD_CS, custom_spi, SPI_ClockHz)) {
    Serial.println("Card Mount Failed");
    return;
  }  uint8_t cardType = SD.cardType();

  if (cardType == CARD_NONE) {
    Serial.println("No SD card attached");
    return;
  }

  Serial.print("SD Card Type: ");
  if (cardType == CARD_MMC) {
    Serial.println("MMC");
  } else if (cardType == CARD_SD) {
    Serial.println("SDSC");
  } else if (cardType == CARD_SDHC) {
    Serial.println("SDHC");
  } else {
    Serial.println("UNKNOWN");
  }

  uint64_t cardSize = SD.cardSize() / (1024 * 1024);
  Serial.printf("SD Card Size: %lluMB\n", cardSize);

  listDir(SD, "/", 0);
  createDir(SD, "/mydir");
  listDir(SD, "/", 0);
  removeDir(SD, "/mydir");
  listDir(SD, "/", 2);
  writeFile(SD, "/hello.txt", "Hello ");
  appendFile(SD, "/hello.txt", "World!\n");
  readFile(SD, "/hello.txt");
  deleteFile(SD, "/foo.txt");
  renameFile(SD, "/hello.txt", "/foo.txt");
  readFile(SD, "/foo.txt");
  testFileIO(SD, "/test.txt");
  Serial.printf("Total space: %lluMB\n", SD.totalBytes() / (1024 * 1024));
  Serial.printf("Used space: %lluMB\n", SD.usedBytes() / (1024 * 1024));
}


void loop() {

}

[Deva_Rishi]

@nickr7

I used this code for testing the microSD modul:

Great ! and does a card show up when you run it ?

I mean does it all work ?

What you need to do is create a new function that is almost the same as

void appendFile(fs::FS &fs, const char * path, const char * message)

but that instead of const char * message takes 2 arguments ( or even 1 if you do the millis() reading as part of the function) The timeCode and the midi command (mind you, this could also be 3 separate arguments.)
And write those to the file instead.

Then basically for reading from the file you need to do the same.

If you want to use

@nickr7

#include "StkAll.h"

You may have to use the specific midi libary that they specify on the github repository, but that may not be required.

Midi reception on the ESP32 is a little tricky using the standard MIDI.h Only UART2 works properly straight out of the box since UART0 is directly connected the USB to TTL converter which disturbs reception. If you want to assign alternate pins to UART2, you can but you should do so after MIDI.begin(), since MIDI.begin just calls Serial.begin() which causes the pin assignment to it's default pins.

I actually just modified serialMIDI.h

    void begin()
	{
        // Initialise the Serial port
        #if defined(AVR_CAKE)
            mSerial. template open<Settings::BaudRate>();
        #else
            mSerial.begin(Settings::BaudRate);
        #endif
	}

into

    void begin()
	{
        // Initialise the Serial port
		#if defined (ARDUINO_ARCH_ESP32) || defined(ESP32)
		// easiest is to manually call begin() for an ESP32 and then include the RX & TX pins
		// Serial1 default pins can not be used, but i use rx=5 & tx=18
		// Serial defaultpins are connected to USB, RX can not be used, may as well just use rx=15 & tx=4
		// Serial2 default pins are fine to use, but may as well be declared rx=16 & tx=17
		#elif defined(AVR_CAKE)
            mSerial. template open<Settings::BaudRate>();
        #else
            mSerial.begin(Settings::BaudRate);
        #endif
	}

and call Serial.begin() from within the sketch, which also allows me to use UART1 which requires alternate pins to be assigned or it crashes (on me it does anyway)

[nickr7]

@Deva_Rishi

Great ! and does a card show up when you run it ?

I mean does it all work ?

Yes, partially, but also partially no. It's unable to open the directory. The output:

15:51:42.036 -> Setup-Start
15:51:42.036 -> custom_spi.begin(SCK=18, MISO=19, MOSI=23, SD_CS=15 done
15:51:42.036 -> Mounting SD-Card
15:51:42.036 -> SD.begin(SD_CS_pin=15custom_spi SPI_ClockHz=80000
15:51:42.318 -> SD Card Type: SDHC
15:51:42.318 -> SD Card Size: 3781MB
15:51:42.318 -> Listing directory: /
15:51:42.365 ->   FILE: test.txt  SIZE: 0
15:51:42.365 ->   FILE: foo.txt  SIZE: 13
15:51:42.785 ->   DIR : System Volume Information
15:51:43.429 ->   DIR : MidiFiles
15:51:43.982 ->   DIR : MP3Files
15:51:43.982 -> Creating Dir: /mydir
15:51:48.881 -> Dir created
15:51:48.881 -> Listing directory: /
15:51:48.970 ->   FILE: test.txt  SIZE: 0
15:51:48.970 ->   FILE: foo.txt  SIZE: 13
15:51:48.970 ->   DIR : mydir
15:51:49.387 ->   DIR : System Volume Information
15:51:50.029 ->   DIR : MidiFiles
15:51:50.585 ->   DIR : MP3Files
15:51:50.585 -> Removing Dir: /mydir
15:51:51.055 -> Dir removed
15:51:51.055 -> Listing directory: /
15:51:51.100 ->   FILE: test.txt  SIZE: 0
15:51:51.100 ->   FILE: foo.txt  SIZE: 13
15:51:51.573 ->   DIR : System Volume Information
15:51:51.573 -> Listing directory: System Volume Information
15:51:51.573 -> Failed to open directory
15:51:52.210 ->   DIR : MidiFiles
15:51:52.210 -> Listing directory: MidiFiles
15:51:52.210 -> Failed to open directory
15:51:52.723 ->   DIR : MP3Files
15:51:52.723 -> Listing directory: MP3Files
15:51:52.723 -> Failed to open directory
15:51:52.723 -> Writing file: /hello.txt
15:51:53.834 -> File written
15:51:54.347 -> Appending to file: /hello.txt
15:51:54.440 -> Message appended
15:51:54.581 -> Reading file: /hello.txt
15:51:54.581 -> Read from file: Hello World!
15:51:54.628 -> Deleting file: /foo.txt
15:51:54.952 -> File deleted
15:51:54.952 -> Renaming file /hello.txt to /foo.txt
15:51:55.326 -> File renamed
15:51:55.326 -> Reading file: /foo.txt
15:51:55.326 -> Read from file: Hello World!
15:51:55.373 -> 0 bytes read for 0 ms
15:53:11.333 -> 1048576 bytes written for 75940 ms
15:53:11.616 -> Total space: 3773MB
15:53:11.616 -> Used space: 8MB

[Deva_Rishi]

@nickr7

Yes, partially, but also partially no. It's unable to open the directory.

How did you create the directory ?

Have you tried by adding

uint64_t cardSize = SD.cardSize() / (1024 * 1024);
  Serial.printf("SD Card Size: %lluMB\n", cardSize);

  listDir(SD, "/", 0);
  createDir(SD, "/midiDir");  // adding it here (might be wise to check if it exists or you'll make many versions of it.
  createDir(SD, "/mydir");
  listDir(SD, "/", 0);
  removeDir(SD, "/mydir");

[nickr7]

@Deva_Rishi

How did you create the directory ?

Have you tried by adding

Yes, I have also tried that.

I also double-checked if all the permissions of the SD card are allowed.

But nothing changed.

[Deva_Rishi]

@nickr7

But nothing changed.

That's weird, because if the sketch can create & delete '/mydir' it can basically do that with any directory name as long as it starts with a '/'

[nickr7]

@Deva_Rishi

That's weird, because if the sketch can create & delete '/mydir' it can basically do that with any directory name as long as it starts with a '/'

I dont understand it too

[Deva_Rishi]

Well it works just fine for me.

[nickr7]

How does the main directory of your SD card look like? I mean what are the names of the folders?