File permissions of custom board packages

I'm working on a custom board package that needs additional executables for the compile and upload process (from the directory with the platform.txt file the executables live in the tools/osx and tools/win directories). We have everything working fine for PC, but on mac the executables are missing their execute permission when Arduino unzips the package archive. We can manually run chmod +x on each executable to solve the issue, but would love to avoid that extra step.

I have been doing all my testing using the 1.8.19 version of the IDE. Interestingly, on the release candidate build of the 2.0 IDE (2.0.0-rc3) this isn't an issue. The executables don't lose their execute permission bit when the package is unzipped by the new IDE.

Any ideas for solving this issue or information on how the two IDEs' unzipping steps may differ would be much appreciated!

Hi @rjpaetz.

Is there a specific reason for putting them there rather than installing them using the normal Boards Manager tools dependencies system?

This is the approach taken by most board packages and there don't seem to be any problems with file permissions for the many thousands of macOS users of those packages.

I know that ESP8266 and ESP32 do put tools inside the platform itself, but it looks like they use Python scripts when installed on a non-Windows hosts:

Arduino IDE 2.x uses Arduino CLI for all non-GUI operations. It is using the gRPC interface, but the arduino-cli core install commandof the command line interface should be equivalent if you want to experiment at a lower level.

Arduino IDE 1.8.19 also uses Arduino CLI (still wrapped in the shell of its origin in arduino-builder) for some things, but not Boards Manager at this time. The original design of the IDE implemented everything in the Java code base and this has gradually been replaced component by component with the Go-based tooling with the goal of leaving behind only the GUI-specific parts in the IDE code base. That has been achieved in Arduino IDE 2.x, but the transition was not complete in the classic Arduino IDE so you end up with some of these behavioral mismatches between the classic IDE and IDE 2.x/Arduino CLI/Arduino Web Editor.

Thanks for such a quick response @in0!

Not a very satisfying answer, but when we were first setting up our package we were following an example that didn't use the tools dependencies system. However, after digging into it more it's clearly the way to go, at least for most of our executables.

The one exception is a shell script that we were also having permissions issues with. In the documentation it sounds like the tools dependencies system is explicitly for binary executables, so I'm assuming we couldn't include the script?

Luckily, I think the ESP32 platform shows a way around that issue. Instead of running the shell file directly, we can run it through sh. This mimics how the ESP32 platform runs its python scripts using python script.py instead of ./script.py.
Our recipe using the shell script would change from:
recipe.hooks.objcopy.postobjcopy.4.pattern.macosx="{runtime.platform.path}/tools/osx/patch_output.sh" ...
to:
recipe.hooks.objcopy.postobjcopy.4.pattern.macosx= sh "{runtime.platform.path}/tools/osx/patch_output.sh" ...

(I found a Unix Stack Exchange post which explains why this works for those who are interested)

However, I still have two questions about the tools dependencies system I couldn't find in the documentation:

  • Is there a list of all possible values for the host field that determines which flavor of build is downloaded by the Boards Manager? Specifically I was wondering what the value is for M1 macs?
  • Relatedly, would we need to provide binaries specific to the M1 mac at all? Or would the Boards Manager just download the build flavor for the x86_64-apple-darwin14 host since the Arduino IDE runs under Rosetta (at least I assume it does?)

Also, thank you for the great info on the differences between the classic IDE and the IDE 2.x!

You can use it for anything. It just extracts the archive to a folder based on the packager, tool name, and version during the Boards Manager installation. I think the reason the Espressif platforms bundle some of the tools is to facilitate beta testing, where you are just cloning their repository to your computer and so don't have Boards Manager to automagically install tools for you.

I have also noticed this ambiguity in the Arduino Package Index Specification. It gives some examples of valid host values, but no comprehensive list.

You can see the algorithm, and perhaps why there is no comprehensive list, here:

https://github.com/arduino/arduino-cli/blob/0.20.2/arduino/cores/tools.go#L126-L195

That is correct. The code linked above will give preference to an available tool with packages[].tools[].systems[].host value matching arm64-apple-darwin.* regex, but falls back to x86_64-apple-darwin.* and then to i[3456]86-apple-darwin.* if there is not tool for that host.

You are welcome. I'm glad if I can be of assistance.