Specifying a custom cores directory from the command line or the config file.

[obra]

Hi!

I have a moderately complex custom build system I’m in the process of migrating from arduino-builder to arduino-cli and I’ve found one bit of configuration that’s absolutely stumping me.

We have our own custom arduino cores which, especially for pre-release code, are inconvenient to package for the boards manager. We’d much rather be able to check them out of git for testing.

With arduino-builder, I could use -hardware to specify a directory containing my own cores directory containing {avr,someotherarch}/{boards.txt,platform.txt,libraries/,variants/} etc.

With arduino-cli, it looks like the best I can do is to specify ARDUINO_DIRECTORIES_USER to one level -up- from that directory. Is that right?

Is there some other configuration option that I can pass to arduino-cli to tell it where a custom hardware/ directory is?

Thanks!

[pert]

I'm not aware of an equivalent in Arduino CLI to arduino-builder's -hardware option.

It's not clear to me why you aren't able to checkout the cores under the hardware subfolder of ARDUINO_DIRECTORIES_USER. Would you mind helping me to understand better?

[obra]

Thanks for the very quick response.

It's not clear to me why you aren't able to checkout the cores under the hardware subfolder of ARDUINO_DIRECTORIES_USER. Would you mind helping me to understand better?

That, or a symlink farm for the various components is, indeed, my likely workaround, but it feels like a hack. Were this a one-shot "how do I build my sketch" question, I'd have done that without blinking.

The additional context is that we make make computer keyboards that run arduino-derived firmware. We have our own libraries and our own core, as well as extensive command-line tooling for folks who don't want to use the Arduino IDE. That predates arduino-cli and we're in the process of modernizing it, now that arduino-cli seems to have stabilized. I'm very excited about all the stuff we'll no longer need to do ourselves.

Some end-users do indeed just use the packaged versions of our core and libraries that we publish for installation using the boards manager. But many others prefer to run the current codebase directly from git, and I'm working on making that as straightforward and seamless as possible. With arduino-builder, we could at least construct things such that a checkout started at the directory that would be dropped into 'hardware/keyboardio' in a sketchbook. I was very much looking forward to being able to do something less hacky with arduino-cli.

Forcing a single directory structure feels like a strangely limiting design choice if it's intentional, and a step backward from arduino-builder in terms of flexibility.

I realize now that I'm actually going to end up with a similar issue around --libraries: I'd really like to be able to include individual libraries in the search path, without having to stick them one extra level deeper in the directory structure so that --libraries /path/to/my-library's-parent-directory/ doesn't attempt pick up ever sibling directory of my library's checkout as a possible library to include.

What I'm really looking for, if this is not indeed possible right now, is some read on whether it's a design decision or just something not-yet-implemented.

[obra]

For context: https://github.com/keyboardio/Kaleidoscope is the project.

The wrapper tool we've been using for the last 4? years to make commandline builds realtively straightforward is https://github.com/keyboardio/Kaleidoscope/blob/master/bin/kaleidoscope-builder

I want to be very clear that I'm looking forward to retiring as much of this code as possible ;)

[pert]

obra: I realize now that I'm actually going to end up with a similar issue around --libraries: I'd really like to be able to include individual libraries in the search path, without having to stick them one extra level deeper in the directory structure so that --libraries /path/to/my-library's-parent-directory/ doesn't attempt pick up ever sibling directory of my library's checkout as a possible library to include.

I don't believe arduino-builder has that capability. Am I wrong?

Arduino CLI's arduino-cli compile --libraries provides the same functionality as arduino-builder -libraries: https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/#options

It looks like you have libraries bundled with your core though, so it's not clear to me why it would be necessary to specify custom library folders.

obra: What I'm really looking for, if this is not indeed possible right now, is some read on whether it's a design decision or just something not-yet-implemented.

I think the best way to proceed is to open a feature request in the Arduino CLI repository's issue tracker.

I would suggest providing a clear example of why you need it including the example folder structure you want to create. I'm pretty knowledgeable on this subject and am really making an effort (but I'll confess I didn't study the 700 line script) to understand your use case, but still I can't understand why you need it.

That's not to say I'm opposed to adding it. It especially makes sense in the context of Arduino CLI already having an arduino-cli compile --libraries option. I just think it never hurts to present a compelling argument when making a feature request. Even if the feature would be added regardless, there's always the question of when it gets added, and a strong argument might bump that work up in the "to-do" list.

[obra]

pert: I don't believe arduino-builder has that capability. Am I wrong?

Arduino CLI's arduino-cli compile --libraries provides the same functionality as arduino-builder -libraries: https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/#options

You are not at all wrong. The libraries thing is absolutely not a regression.

pert: It looks like you have libraries bundled with your core though, so it's not clear to me why it would be necessary to specify custom library folders.

Use cases include building against a different core or building a copy of a library from a different checkout against the core.

pert: I think the best way to proceed is to open a feature request in the Arduino CLI repository's issue tracker.

I would suggest providing a clear example of why you need it including the example folder structure you want to create. I'm pretty knowledgeable on this subject and am really making an effort (but I'll confess I didn't study the 700 line script) to understand your use case, but still I can't understand why you need it.

That's not to say I'm opposed to adding it. It especially makes sense in the context of Arduino CLI already having an arduino-cli compile --libraries option. I just think it never hurts to present a compelling argument when making a feature request. Even if the feature would be added regardless, there's always the question of when it gets added, and a strong argument might bump that work up in the "to-do" list.

I appreciate the quick and thoughtful responses late on a Saturday night. At the core, I think what I'm trying to figure out is about whether the directory structures supported by arduino-cli are intentionally "opinionated." If Arduino wants developers to keep their code organized in a certain way, I can understand that.

Before opening the feature request, I'm going to spend my time trying to do things the way arduino-cli wants users to do them.

I apologize for doing a poor job explaining my use case here. I'd posted the question to the forum, rather than as a feature request, because I was trying to figure out if I was just missing docs for an existing feature.

[pert]

obra: I think what I'm trying to figure out is about whether the directory structures supported by arduino-cli are intentionally "opinionated." If Arduino wants developers to keep their code organized in a certain way, I can understand that.

Arduino's trajectory from Arduino IDE to arduino-builder to Arduino CLI is pretty interesting to me. Originally, everything was crammed into the Arduino IDE codebase. The IDE didn't even have a CLI up until somewhere around 2013. I think it really became clear to Arduino that this needed to change when they started work on the Arduino Web Editor. So they replaced the build code in the Arduino IDE with arduino-builder. This was the first step to making the Arduino IDE's Java codebase be purely about the GUI, however, there were still a lot of non-GUI code left behind (e.g., Library/Boards Manager) that wouldn't really fit in with the stated purpose of arduino-builder. I guess they came to the conclusion that the best approach was to put all the non-GUI code in a single tool, rather than having a separate tool for each component. So Arduino CLI is almost a compromise between the approach taken for the original Arduino IDE and that taken for arduino-builder.

Arduino IDE is beloved by the average user, but not at all well suited to integration with other tools or automation applications such as CI. arduino-builder is a flexible, single purpose tool, but very few people other than Arduino found any use for it.

So I think the largest demand is for a tool that provides the functionality of the Arduino IDE in a command line tool. However, Arduino CLI was also designed to be the engine under the hood in all of Arduino's official development tools (Arduino IDE, Arduino Web Editor, Arduino Pro IDE), and there's no reason it can't fill the same role in 3rd party applications as well. In addition to the CLI, it has a gRPC interface or can be used as a Golang module. So there is a need to find a balance between keeping it user friendly, but also being capable for advanced use cases. Unlike the Arduino IDE, Arduino CLI is not targeted to beginners, so we aren't so constrained in how much complexity we can expose in the UI. However, Arduino's core value is making complex things as easy as possible, so it is important to design the UI carefully and avoid adding unnecessary complexity.