Storing References to a File on SD Card

Using Arduino Storage
1

Hello everyone!
Long time watcher, first time poster. Hopefully I can articulate my problem well, feel free to give me pointers on posting here.

I am working on a project built on the ESP32-S3 board, using the latest version of the Arduino core for ESP32 (v3.0.0-alpha3). This project is an MP3 player with a file browser UI. File in question attached below. The entire project is quite large and most of it is not relevant to this question, I can attach full source if needed. This is very much still WIP and I am aware that there are parts of this UI implementation that need review. I am just focused on this file reference problem right now.

#ifndef UI_H
#define UI_H

#include <Adafruit_GFX.h>
#include <Fonts/TomThumb.h>

#include "display.h"
#include "player.h"
#include "buttons.h"
#include "button_functions.h"
#include "src/track_tags/tags.h"

const int MAX_DISPLAY_ITEMS = 12;
const int MAX_DISPLAY_INDEX = MAX_DISPLAY_ITEMS - 1;
const int MAX_BROWSER_ITEMS = 1024;

const int FILE_TYPE_MUSIC = 0;
const int FILE_TYPE_DIRECTORY 1;
const int FILE_TYPE_PLAYLIST 2;

struct UiFileEntry{
  char path[256];
  char name[128];
  uint8_t type;
};

class UI {
  public:
    Adafruit_SSD1327* display;
    Player* player;
    Buttons* buttons;
    char cwd[128] = "Initializing...";
    bool pendingUpdate = false;
    UI(Adafruit_SSD1327* _display, Player* _player, Buttons* _buttons) {
      display = _display;
      player = _player;
      buttons = _buttons;
    }

    bool init(){
      // Allocate strings for UI display in PSRAM
      for (int i=0; i < MAX_BROWSER_ITEMS; i++){
        browserList[i] = (UiFileEntry *)ps_malloc(sizeof(UiFileEntry));
      }
      display->setRotation(3);
      display->setTextWrap(false);
      enterDirectory("/");
      displayStatusBar();
      displayNavBar();
      displayCwdBar();
      displayMain();
      // TODO: Playqueue should not start prepopulated, remove once dev work is far enough along
      for(int i=0; i<browserLen; i++){
        player->appendQueueLast(browserList[i]->path);
      }
      buttons->clickWheel.setUpFunction(UI_SCROLL_UP);
      buttons->clickWheel.setDownFunction(UI_SCROLL_DOWN);
      buttons->clickWheel.setShiftUpFunction(UI_PAGE_UP);
      buttons->clickWheel.setShiftDownFunction(UI_PAGE_DOWN);
      buttons->next.setPressFunction(UI_SCROLL_UP);
      buttons->previous.setPressFunction(UI_SCROLL_DOWN);
      buttons->select.setPressFunction(UI_SELECT);
      buttons->play.setPressFunction(UI_PLAY_PAUSE);
      buttons->option.setPressFunction(UI_DIRECTORY_UP);
      return true;
    }

    void tick(){
      if(pendingUpdate){
        Serial.println("Updating display");
        display->display();
        pendingUpdate = false;
      }
    }

    void scroll(int items) {
      selectionIndex = selectionIndex + items;
      if (selectionIndex + displayStartIndex < 0) {
        // Beginning of browser reached
        selectionIndex = 0;
      } else if (selectionIndex + displayStartIndex >= browserLen) {
        // End of browser reached
        selectionIndex = MAX_DISPLAY_INDEX;
      } else if (selectionIndex < 0) {
        displayStartIndex = max(0, selectionIndex + displayStartIndex);
        selectionIndex = 0;
      } else if (selectionIndex > MAX_DISPLAY_INDEX){
        // Move entire display list down
        displayStartIndex = min(browserLen - MAX_DISPLAY_ITEMS,
                                  displayStartIndex + selectionIndex - MAX_DISPLAY_INDEX);
        selectionIndex = MAX_DISPLAY_INDEX;
      }
      Serial.print("Cursor index: "); Serial.println(selectionIndex);
      Serial.print("Browser index: "); Serial.println(getBrowserIndex());
      Serial.println();
      displayMain();
    }

    void select() {
      UiFileEntry *selectedItem = browserList[displayStartIndex + selectionIndex];
      if (selectedItem->type == FILE_TYPE_MUSIC) {
        player->appendQueueLast(selectedItem->path);
        Serial.print("Appending "); Serial.println(selectedItem->name);
      } else if (selectedItem->type == FILE_TYPE_DIRECTORY) {
        enterDirectory(selectedItem->path);
        displayMain();
        displayCwdBar();
      } else {
        Serial.println("Selected item is not implemented yet");
      }
    }

    void upDirectory() {
      if (strcmp("/", cwd) == 0) {
        // We are at the root directory, take no action
        return;
      }
      char * const last = strrchr(cwd, '/');
      if (last == cwd) {
        // We are one away from root, do not blow away trailing /
        cwd[1] = '\0';
      } else if (last != NULL){
        *last = '\0';
      }
      enterDirectory(cwd);
      displayMain();
      displayCwdBar();
     }

  private:
    UiFileEntry *browserList[MAX_BROWSER_ITEMS];
    int browserLen = 0;
    const byte navStartY = 115;
    const byte boxWidth = SCREEN_WIDTH / 4;
    const byte boxHeight = SCREEN_HEIGHT -  navStartY;
    const byte navTextY = navStartY + 9;
    const byte mainStartY = 17;
    const byte cwdStartY = 9;
    uint16_t displayStartIndex = 0;
    int8_t selectionIndex = 0;
    uint8_t rowStartYVals[MAX_DISPLAY_ITEMS] = {16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104};

    int getBrowserIndex() {
      return displayStartIndex + selectionIndex;
    }

    void displayStatusBar() {
      display->drawFastHLine(0, 7, SCREEN_WIDTH, SSD1327_WHITE);
      display->drawFastHLine(0, 8, SCREEN_WIDTH, SSD1327_WHITE);
      display->setFont(&TomThumb);
      display->setTextColor(SSD1327_WHITE);
      display->setTextSize(1);
      display->setCursor(0, 5);
      display->print("Whaddup");
      display->setCursor(112, 5);
      display->print("100%");
    }

    void displayNavBar() {
      display->setFont(&TomThumb);
      display->setTextColor(SSD1327_WHITE);
      display->setTextSize(1);
      for (int i=0; i<4; i++){
        display->drawRoundRect(i * boxWidth, navStartY, boxWidth, boxHeight, 4, SSD1327_WHITE);
      }
      display->setCursor(3, navTextY); display->print("Browser");
      display->setCursor(38, navTextY); display->print("Queue");
      display->setCursor(68, navTextY); display->print("Playing");
      display->setCursor(98, navTextY); display->print("Settings");
      pendingUpdate = true;
    }

    void displayCwdBar() {
      display->fillRect(0, cwdStartY, 128, 7, SSD1327_WHITE);
      display->setFont(&TomThumb);
      display->setTextColor(SSD1327_BLACK);
      display->setCursor(2, 14);
      display->print(cwd);
      pendingUpdate = true;
     } 

    void displayMain() {
      display->fillRect(0, mainStartY, 128, navStartY - mainStartY, SSD1327_BLACK);
      // TODO: This rectangle will eventually be the cursor
      display->fillRect(0, rowStartYVals[selectionIndex], 128, 9, 2);
      display->setFont();
      display->setCursor(0, mainStartY);
      display->setTextColor(SSD1327_WHITE);
      for (int i = displayStartIndex; i < min(browserLen, MAX_DISPLAY_ITEMS) + displayStartIndex; i++){
        display->println(browserList[i]->name);
      }
      pendingUpdate = true;
    }

    void enterDirectory(char *path) {
      browserLen = 0;
      File dir = SD_MMC.open(path);
      strcpy(cwd, path);
      while(true) {
        File entry = dir.openNextFile();
        if (!entry) {
          // We've reached the end of the directory
          break;
        } else if (!strncmp(entry.name(), ".", 1)) {
          // Files starting with "." are hidden, do not display
          continue;
        }
        if (entry.isDirectory()) {
          strcpy(browserList[browserLen]->name, entry.name());
          strcpy(browserList[browserLen]->path, entry.path());
          browserList[browserLen]->type = FILE_TYPE_DIRECTORY;
          // TODO: Add a visual indicator that this is a directory
        } else {
          getTrackTitle(browserList[browserLen]->name, entry);
          strcpy(browserList[browserLen]->path, entry.path());
          browserList[browserLen]->type = FILE_TYPE_MUSIC;
        }
        browserLen++;
        entry.close();
      }
      dir.close();
    }

};

#endif

I am currently storing an array of structs representing the contents of a single directory in RAM. I would like this in working memory to help keep the UI quick and responsive. Currently I store the filepath, a display name (track title or directory name) and a type descriptor for each file.

struct UiFileEntry{
  char path[320];
  char name[128];
  uint8_t type;
};

The browser display array is currently 1024 items long, so as you can imagine this has a pretty big memory footprint. This is not giving me problems as I have 8MB of PSRAM available, but I can't help but feel like there is a more memory efficient way to store reference to a file.

I've tried storing a pointer to the File object in my UIFileEntry struct, as the File object is only 24 bytes, but it is not something I can work with. I've looked at doing something like below:

File testFile = SD_MMC.open("/text.txt");
Serial.println(testFile.read());
testFile.close();
testFile.open();
Serial.println(testFile.read());
testFile.close();

But the File object has no means for reopening a file that has been closed.
I've also looked at something like this:

File testFile = SD_MMC.open("/text.txt");
Serial.println(testFile.read());
testFile.close();
testFile = SD_MMC.open(testFile.path());
Serial.println(testFile.read());
testFile.close();

But that results in instant core dump. Given that there is a max open file limit in the SD library, I assume there is a pool of memory not contained in the FIle object that is no longer accessible once the file is closed, as it is freed up for a new open file. Correct me if I am wrong.

Based on my limited understanding of filesystems, the path an abstract that should resolve to something more concrete like a starting address in memory for the file or something. Is there something smaller than the path string that can be used to reference the file? Alternatively, is there some way to reuse the File object once it is closed that I have passed over?
The current implementation is functional and still leaves me with 90% of my RAM, but my gut says there is a better way to implement this.

2

Welcome to the forum

char path[320];

Wow, that is a very long path

3

Thank you!
Haha yes it is. This is not the longest path in my library, but an example of how this can happen:

/music/Radiohead/A Moon Shaped Pool/01-10 Tinker Tailor Soldier Sailor Rich Man Poor Man Beggar Man Thief.mp3

This is how my library organization tool builds it, and makes for an easy structure to navigate manually. If I can use the same structure for the player, it means loading music on the SD card is a simple rsync script on the computer side

Using a find | wc -L operation on the library, I saw the maximum path length is 260, so I decided it pad that number out a little bit

4

If the object could be reused, it means that it still to occupy the same memory as before the closing.

It is not clear for me, how the pointer to UIFileEntry could help you to free the memory? To be valid the pointer must point to real memory location. If you save the pointer, but delete the structure itself, the pointer became useless.

5

Oh, yes I think I misspoke, the idea would be to replace

struct UiFileEntry{
  char path[320];
  char name[128];
  uint8_t type;
};

with

struct UiFileEntry{
  File fileObject;
  char name[128];
  uint8_t type;
};

How does one reuse the file object? File itself does not have an open() method, I've dug around the SD and FS libraries a bit and I don't see an obvious way of doing this.

6

I think this file system organization is not good. Firstly, very long file names, and secondly, many music players and file libraries do not work with spaces in file names.
By the way, what kind of library are you using? If there is a limit on the length of the file name?
I would advise renaming the files so that there are no spaces in the names and that the name length does not exceed some reasonable length - for example 25-30 characters

7

I'm using beets.io to organize the library. It stacks everything in a Artist/Album/Track directory structure and uses <Disc Number>-<Track Number><Track Title>.<File extension> for filenames
Its a pretty universally usable structure. The arduino code provided above also works without issue, I'm just trying to optimize the memory footprint at this point.
My music library is currently synced across a few devices, I'd rather not modify it if I don't have to

8

Is it works on ESP32 ?
What the library do you use on ESP32 to work with file system?

9

beets runs on the device that keeps the music library upstream copy, it just enforces the directory structure and naming convention, I can then sync it onto an SD card that I plug into the ESP32. The ESP32 SD_MMC library is what is accessing the filesystem
I believe SD_MMC has a near-identical API to Arduino's SD it just allows for hardware interfacing with an SD card using 4-bit DIO instead of SPI

10

Arduino SD library uses short 8.3 filenames and do not accept spaces. SD_MMC has an option to use "long filenames", but the maximum is 255 characters for entire path, not the name only.

11

I see, I probably haven't run into that 260 character filepath yet then

12

so you path field could be shorter

13

Yeah, I think that one track name is a significant outlier, I could manually clip that and see if I can make a rule in beets to truncate the filename if it is over a certain length.
My curiosity remains about using an implementation that does not require storing filepaths though. You had mentioned being able to reopen a File object after closing it. Did I understand that right?

14

No, I didn't meant that

15

Okay, I appreciate your insight on file structure. Thanks!

16

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.