Best Practices for Managing Configurations in Arduino Projects (Using JSON or Alternative Approaches)

Hello everyone,

I’m currently working on a project using an Arduino Micro that involves managing multiple “layers” of key mappings for a custom MIDI controller. Each layer has an array of key mappings, and each key can trigger either a keyboard press or a MIDI note.

My background is in web development, where I would typically manage such configurations using JSON files, which allows for easy scalability and configurability. However, since the Arduino Micro doesn’t have an SD card slot or built-in file storage, I’m looking for advice on the best practices for handling this kind of configuration in an Arduino environment.

My Requirements:

Scalability: I want to be able to easily add, modify, or remove layers and key mappings.

Configurability: Ideally, I would like to keep the configuration separate from the main logic to keep the code clean and modular.

Memory Efficiency: Since the Arduino Micro has limited memory, I want to avoid consuming too much RAM.

UX: In an ideal scenario, I'd create a web interface where the user could configure their own mappings and that config would be stored in the arduino.

Current Approach:

I am considering placing my configuration data (the layers and key mappings) in a separate header file (config.h). Here’s a simplified version of what I’m doing:

// config.h

#define MAX_TARGETS 6
#define NUM_LAYERS 2

struct KeyMap {
  int pin;
  const char* action;
  int targets[MAX_TARGETS];
  int targetCount;
  const char* label;
};

struct Layer {
  const char* name;
  KeyMap keys[6];
};

Layer layer1 = {
  "1.Keys", 
  {
    {2, "keypress", {'a'}, 1, "A"},
    {3, "keypress", {'b'}, 1, "B"},
    // Other keys...
  }
};

Layer layer2 = {
  "2.Notes", 
  {
    {2, "keypress", {65}, 1, "65"},
    {3, "keypress", {66}, 1, "66"},
    // Other keys...
  }
};

Layer* layers[NUM_LAYERS] = {&layer1, &layer2};

My Questions:

  1. Is using a header file to store configuration data a common and recommended approach for Arduino projects?

  2. Are there any alternative or better approaches for managing such configurations in an Arduino environment?

  3. What considerations should I keep in mind regarding memory usage and scalability when handling configurations this way?

  4. Would there be any significant benefits to using something like EEPROM or PROGMEM for storing configurations, or would that add unnecessary complexity?

  5. Is there a common pattern to be able to create the config files in a web app and deploy them to the arduino without having to install the arduino IDE and without the need of a SD card?

Any advice or insights you could provide would be greatly appreciated! Really all I'm trying to find is perspective and different approaches. I'm not looking for very precise answers, so anything helps.

Thanks in advance!

Yes. Typically a config.h type file should only contain simple symbol definitions

/*
 * config.h
 */
#define LAYER1 1       // include layer 1
#define ALTKEYS 1      // alternate keyboard mapping.
#define NKEYS  20      // number of keys
#define MAX_TARGETS 6

And actual code, like your structure definitions, would be off in a separate file, keeping the "configuration" and the code that actually implements the configuration separate.

/*
 * config.cpp
 */
#ifdef LAYER1

struct Layer {
  const char* name;
  KeyMap keys[NKEYS];
};

Layer layer1 = {
  "1.Keys", 
  {
    {2, "keypress", {'a'}, 1, "A"},
    {3, "keypress", {'b'}, 1, "B"},
    // Other keys...
  }
#endif

The optimizing compiler will do a good job of omitting unused code and data, and it IS a compiler, so non-code (like structure type definitions) have no effect on the runtime file.
So, if you have a #define FOO 0 in your config file, and code like

   if (FOO) {
      // a lot of code
   }

then all that code will be discarded. This can result in more readable code than the sprinkling of #ifdef blocks that you frequently see, but it can be less obvious that things are omitted...

Well, it would permit you to change configurations without recompiling the code, if that's an advantage. It does add complexity and consumes resources, and you'd need a tool for changing the EEPROM or PROGMEM content.

Not that I know of.

yes

An alternative would be to store data in the EEPROM.

explain your usecase regarding scalability. Do you deploy 2, 20, 200, 2000 devices to how many people and how many devices will a typical user have?

EEPROM: could be changed without reprogramming the target
PROGMEM: less SRAM usage

use a microcontroller with wifi, for example something with an esp8266 or esp32, introduce a webserver with a GUI and store the data in "preferences"

+1 on the idea of switching to an ESP32 or WiFi enabled MKR arduino given your requirements.

It’s about the same size and so much more capable to meet your needs (files, WiFi, BT, OTA update, Serial ports,…)

Thank you so much for all your answers!

Thanks! Yeah, I think I need to investigate this more. It is def an advantage to be able to change configurations without having to compile again. The only missing piece would be how to let users changing the config without installing the arduino ide.

I was not thinking about amount of devices. I was thinking more about adding more features or improving the existing ones or even just being able to store more layers.

This is an awesome idea. I will def try with an ESP32, maybe even with a battery to go fully wireless. And the web server idea is great. I didn't think about that.

My original idea was something like this:
https://launcher.keychron.com/#/keymap

It's a web ui that let's users connect a keyboard and remap it's keys. That's exactly what I want to do. I don't know if there will ever be other users, but even just for me, that's the experience I want to build. I plan to keep changing the config of the project very often and I want the process to be as smooth as possible.