Platform Specification Question - is it a bug in Arduino IDE?

ekawahyu · September 12, 2020, 9:39pm

I am following this guideline to develop and modify existing code:

https://arduino.github.io/arduino-cli/platform-specification/

Folder structure:

Arduino > hardware > vendor > variant
Arduino > hardware > vendor > tools

Before developing my own board, I am testing this for 2 existing vendors with 1 variant each, with folder structure that look like these:

Arduino > hardware > sandeepmistry > nRF5
Arduino > hardware > sandeepmistry > tools > gcc-arm-none-eabi
Arduino > hardware > sandeepmistry > tools > openocd

Arduino > hardware > STM32 > stm32
Arduino > hardware > STM32 > tools > xpack-arm-none-eabi-gcc
Arduino > hardware > STM32 > tools > STM32Tools

I can build the blink sketch for both platform, but I am not able to upload it at all. Arduino IDE does not seem to find the location of the tools.

In platform.txt for nRF5, I had to change the programming tool path like this:

#tools.openocd.path={runtime.tools.openocd-0.10.0-dev.nrf5.path}
tools.openocd.path={runtime.hardware.path}/tools/openocd/0.10.0-dev.nrf5

In platform.txt for STM32, I had to change the programming tool path like this:

#tools.stm32CubeProg.path.linux={runtime.tools.STM32Tools.path}/tools/linux
tools.stm32CubeProg.path.linux={runtime.hardware.path}/tools/STM32Tools/1.4.0/tools/linux

Arduino IDE does see runtime.tools just fine, but something is not working in there when being used by the programmer tool. Instead, pointing it to the absolute path and runtime.hardware, it works perfectly. I think it is a bug in the Arduino IDE. Anyone else experience the same thing? Am I missing anything? What's the temporary workaround for this?

My system is Ubuntu 18.04, Arduino IDE 1.8.13

pert · September 12, 2020, 10:30pm

I'm not sure I understand your question. When you say "something is not working in there", is the "something" you are referring to the fact that you need to change {runtime.tools.TOOL_NAME.path} to {runtime.hardware.path}/path/to/tool?

If so, this is normal and expected. The {runtime.tools.TOOL_NAME.path} feature requires the tool dependency to have been installed via Boards Manager. The Arduino IDE doesn't recognize tool dependencies installed in the sketchbook.

It is more convenient to do development work on a platform installed to the sketchbook. What you can do is install a platform that provides the tools dependencies via Boards Manager, but then have a development copy of the platform alone in your sketchbook. If the release version of the platform is also installed, you can modify the vendor folder name to differentiate the installation in the sketchbook from the other copy of the platform.

You can see an interesting discussion of the development strategies of some Arduino platform developers here:

github.com/arduino/Arduino

How to use an (avr) core from git?

opened 01:07PM - 19 Sep 19 UTC

matthijskooijman

Component: Core question arduino-cli arduino-builder

I often work on the AVR core files, so I have a checkout of the ArduinoCore-avr …repository on my system that I want to usually use. Previously, I just had a symlink from `~/Arduino/hardware/arduino/avr` to the `ArduinoCore-avr` checkout and that was used as the AVR core by the IDE. However, I've just updated to the latest git (I'm also running the IDE from git) and this no longer works. I'm not sure what change broke it exactly, since reverting to (what I think was) the previous version does not unbreak it (so maybe I had some extra local changes before in e.g. the `build/linux` folder, or maybe a custom arduino-builder version too?). The problem I'm seeing now is that: - Arduino-builder gets `-hardware` options in order "build/linux/work/hardware", "~/.arduino15/packages", "Sketchbook/hardware": https://github.com/arduino/Arduino/blob/6116a8e9206124329e0c8b58e828eb5b83b6055a/arduino-core/src/cc/arduino/Compiler.java#L241-L243 It seems that this also determines the priority, meaning that any bundled AVR-core, or any boards-manager installed core is used before my core from git in my sketchbook. I do not have any boards-manager installed core, so to use my custom core I removed the `build/linux/work/hardware/arduino` folder, which makes arduino-builder use the sketchbook core, but then the IDE seems to no longer know about avrdude: java.io.IOException: Cannot run program "{runtime.tools.avrdude.path}/bin/avrdude": error=2, No such file or directory This is confirmed by `--get-pref`, running `arduino --get-pref 2>/dev/null |grep avrdude` returns nothing. I guess this makes sense: Tools are defined by `.json` files, but I have none (for avr) in my `~/.arduino15` directory. I do have `build/linux/work/hardware/package_index_bundled.json` which defines an avrdude tool, but (I guess) since there is no corresponding AVR core directory (I deleted it), the tool is not loaded. I tried: - Putting a symlink to the git avr core in `build/linux/work/hardware/arduino/avr`, which works, but only if the version number in the `package_index_bundled.json` file matches the version number in the core's `platform.txt`. If not, the avrdude is not made available. Since I sometimes jump around different versions, this is not really convenient. - Keeping the original `build/linux/work/hardware/arduino/avr` directory, but giving my core from git a higher version number in its platform.txt (hoping that arduino-builder would use the highest version number), but that did not work (avrdude works, but arduino-builder uses the bundled avr core). - Installing an avr core with the boards manager and replace `build/linux/work/hardware/arduino/avr` with a symlink to the git core. I thought this woujld let the board-manager installed version make its tools available (so avrdude would work), but the symlink in `build/linux/...` has higher priority for arduino-builder, so that core is used. However, I now get `java.io.IOException: Cannot run program "___REMOVE___/bin/avrdude": error=2, No such file or directory`, so apparently it does recognize the tool, but it [has a null path](https://github.com/arduino/Arduino/blob/master/arduino-core/src/processing/app/BaseNoGui.java#L702-L706) somehow... - Keeping the original bundled core and linking to the git core as `~/Arduino/hardware/arduino-git/avr`. This changes the core name and lets the core be installed alongside the regular avr core rather than overriding it (which causes all boards to show up in the boards menu twice). This needs a change in the git core's name in platform.txt to be able to distinguish them and makes the board menu *even longer*, but I guess this is a reasonable way to do this. What is noteworthy is that this problem does not seem to occur in arduino-builder, which has no trouble finding `avr-gcc` (which is also a tool and is also referenced through `runtime.tools.avr-gcc.path`), so it seems the IDE and arduino-builder load their tools differently (but I've never really dug in to this stuff to really understand it...). I guess it comes down to the handling of cores that do not have a corresponding `.json` file to define their dependencies. I guess this also goes for other third-party cores installed directly into the sketchbook, though those typically use the tools from the e.g. the official AVR or SAM cores. Since these third-party cores have a distinct name and offer boards distinct from the official boards, the priority used by arduino-builder is not a problem for them. My `~/Arduino/hardware/arduino-git` approach above essentially turns the git core into a distinct core as well, which is why that works. So I wonder: Are there better ways to support this usecase? If not, is there anything we should or could change to make this easier?

but note that it's no longer necessary to modify the value of the name property for this strategy because a feature was added in Arduino IDE 1.8.13 that creates a separate Tools > Board submenu for the set of boards of the release version and the development version of the platform.

ekawahyu · September 12, 2020, 11:16pm

pert:
I'm not sure I understand your question. When you say "something is not working in there", is the "something" you are referring to the fact that you need to change {runtime.tools.TOOL_NAME.path} to {runtime.hardware.path}/path/to/tool?

If so, this is normal and expected. The {runtime.tools.TOOL_NAME.path} feature requires the tool dependency to have been installed via Boards Manager. The Arduino IDE doesn't recognize tool dependencies installed in the sketchbook.

Let me explain a bit what is working and not working on my setup.

This is where I got the Arduino STM32 Core:

I added the json file under Additional Board Manager URLs so that I can install the package under Board Manager. The installation went smooth, it built just fine and able to upload to one of Discovery board I have.

I'd like to modify the source code and to use my own repo and this is the guide I followed:

Section 3.1. basically says that I can delete the source file (but keeping the installed tools as is) and replace the source file by cloning my own repo. This is tested to work, but I don't like it because I may accidentally delete my repo when it is uninstalled from the Board Manager.

Section 3.2. basically says that I have an option to create my repo under Arduino/hardware (but keeping the installed tools as is). This setup can compile just fine, but it failed to upload with the following error:

{runtime.tools.STM32Tools.path}/tools/linux/stm32CubeProg.sh 0 /tmp/arduino_build_780271/stm32_blink_usbserial.ino.bin -g
java.io.IOException: Cannot run program "{runtime.tools.STM32Tools.path}/tools/linux/stm32CubeProg.sh": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:485)
at processing.app.helpers.ProcessUtils.exec(ProcessUtils.java:11)
at cc.arduino.packages.Uploader.executeUploadCommand(Uploader.java:129)
at cc.arduino.packages.uploaders.SerialUploader.runCommand(SerialUploader.java:383)
at cc.arduino.packages.uploaders.SerialUploader.uploadUsingPreferences(SerialUploader.java:197)
at cc.arduino.UploaderUtils.upload(UploaderUtils.java:77)
at processing.app.SketchController.upload(SketchController.java:732)
at processing.app.SketchController.exportApplet(SketchController.java:703)
at processing.app.Editor$UploadHandler.run(Editor.java:2055)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 11 more
An error occurred while uploading the sketch

ekawahyu · September 13, 2020, 12:11am

By now, somehow, I managed to figure out what {runtime.tools.STM32Tools.path} is pointing to. It is actually looking for the tool under Arduino/hardware folder instead of where it was originally installed by Board Manager. So I made a copy from .arduino15/packages/STM32/tools to Arduino/hardware/STM32/tools. But, Arduino IDE does not seem to see it either. Not until I did this:

#tools.stm32CubeProg.path.linux={runtime.tools.STM32Tools.path}/tools/linux
tools.stm32CubeProg.path.linux={runtime.hardware.path}/tools/STM32Tools/1.4.0/tools/linux

A quick test on {runtime.tools.STM32Tools.path}, it actually points to /home/ubuntu/Arduino/hardware/STM32/tools/STM32Tools/1.4.0

A quick test on {runtime.hardware.path}, it points to /home/ubuntu/Arduino/hardware/STM32

So they both point to the same and correct path. But why only the latter works? That's what I meant with "something is not working in there", because I have no idea how to debug this further without any knowledge of how Arduino IDE works.

pert · September 13, 2020, 8:09am

The only way I can get this error is by deleting ~/.arduino15/package_stm_index.json

The Arduino build system uses the package indexes to generate those {runtime.tools.TOOL_NAME.path} properties, so even though the tool dependency is installed on your hard drive, if that package index file is gone, then there's no property for it. I believe this is the related issue:

ekawahyu:
By now, somehow, I managed to figure out what {runtime.tools.STM32Tools.path} is pointing to. It is actually looking for the tool under Arduino/hardware folder instead of where it was originally installed by Board Manager.

What is your evidence for that? I've spent a lot of time trying to get the Arduino IDE to do that (a few years ago so it's possible the system has changed) without any success. From my investigation, there is no formal support for recognizing tools dependencies under the hardware subfolder of the sketchbook in this way. You can of course provide a path relative to {runtime.hardware.path}, as you discovered, but this makes it difficult to create a boards platform with unique tools dependencies that will work both for manual installation to the sketchbook as well as for installation via Boards Manager.

ekawahyu · September 13, 2020, 12:00pm

Here is my debugging trick. In platform.txt: if you copy {runtime.tools.TOOL_NAME.path} to replace whatever path to the compiler (compiler.path), it is gonna give you an error because you simply point Arduino IDE to location with no compiler in there. But, if you look at the compiler path in this error message and you will see that {runtime.tools.TOOL_NAME.path} is actually pointing to Arduino/hardware.

pert · September 13, 2020, 12:13pm

I wasn't able to reproduce that:

exec: "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\STM32\\tools\\STM32Tools\\1.4.0arm-none-eabi-g++": file does not exist

I'm using Windows because I don't have the STM32 boards platform set up on my Linux machine. So it's possible there's something platform-specific, but it's only of interest to me as a feature if it works on all OSs.

Of course, a bug that only occurs on one OS is still very much a bug and of interest to me. Did you verify that ~/.arduino15/package_stm_index.json is present on your computer?

Juraj · September 13, 2020, 1:19pm

@ekawahyu, install the STM32 boards package in Boards Manager.
then in your boards package or hardware defintion you can refer the tool in boards.txt as STM32:stm32CubeProg

github.com

jandrassy/my_boards/blob/ef85da87a212d490090734c9e58e87bace715aac/stm32/boards.txt#L26


      
          BluePill128k.build.extra_flags=-D{build.product_line} {build.enable_usb} {build.xSerial} {build.bootloader_flags}

          

          BluePill128k.upload.maximum_size=131072

          BluePill128k.upload.maximum_data_size=20480

          BluePill128k.upload.protocol=maple

          BluePill128k.upload.tool=STM32:maple_upload

          BluePill128k.upload.usbID=1EAF:0003

          BluePill128k.upload.altID=2

          

          #########################################################

          BluePill128kswd.name=BluePill F103C8 (128k) SWD

          

          BluePill128kswd.build.vid=0x0483

          BluePill128kswd.build.core=STM32:arduino

          BluePill128kswd.build.mcu=cortex-m3

          BluePill128kswd.build.series=STM32F1xx

          BluePill128kswd.build.board=BLUEPILL_F103C8

          BluePill128kswd.build.product_line=STM32F103xB

          BluePill128kswd.build.variant=STM32:PILL_F103XX

          BluePill128kswd.build.cmsis_lib_gcc=arm_cortexM3l_math

          BluePill128kswd.build.flash_offset=0x2000

ekawahyu · September 13, 2020, 7:44pm

@pert: Yeah, in your machine, the path is pointing to the installation by Board Manager

Of course, a bug that only occurs on one OS is still very much a bug and of interest to me. Did you verify that ~/.arduino15/package_stm_index.json is present on your computer?

@pert: Yep, the json file is still in there. I have been working for few years under Arduino/hardware, but never thought that this issue could be in the Arduino IDE itself. Just recently I thought about it and started to investigate further. I will definitely try to do on both macOS and Windows and see if this issue is not OS related as you said.

ekawahyu · September 13, 2020, 7:56pm

Juraj:
@ekawahyu, install the STM32 boards package in Boards Manager.
then in your boards package or hardware defintion you can refer the tool in boards.txt as STM32:stm32CubeProg

my_boards/stm32/boards.txt at ef85da87a212d490090734c9e58e87bace715aac · JAndrassy/my_boards · GitHub

This solution does not seem to work for me. Do you mind pointing me to the documentation of it? What OS are you on? This is my board.txt:

Disco.menu.upload_method.swdMethod=STM32CubeProgrammer (SWD)
Disco.menu.upload_method.swdMethod.upload.protocol=0
Disco.menu.upload_method.swdMethod.upload.options=-g
Disco.menu.upload_method.swdMethod.upload.tool=STM32:stm32CubeProg

I modified the last line per suggestion, but unfortunately it still gives me the same error.

pert · September 13, 2020, 8:19pm

ekawahyu:
Do you mind pointing me to the documentation of it?

Here it is:
https://arduino.github.io/arduino-cli/latest/platform-specification/#tool-references

ekawahyu · September 14, 2020, 12:09am

Alright, I haven't got this tested on Windows just yet, but this does not seem to me as OS specific. Because I can see the exact same behavior on Linux and macOS.

@pert: your tool path is showing .arduino15/packages//tools because it is the default installation by Board Manager. If you want to do a little experiment and move whatever inside tools folder to Arduino/hardware/vendor/tools, I am pretty sure Arduino IDE will point to this new location. It will compile but fail to upload. The only way to make upload to work is having both variant and tools in the same default location.

Here is the way to reproduce it, in my case I use STM32. I am sure you can try to reproduce this with any platform of your choice and let me know how it goes.

First step, install the package from Board Manager, supply the json file if you cannot find your board. This will create a folder structure as follows:

.arduino15/packages//hardware//
.arduino15/packages//tools/

What we can do now is:

Move the variant folder to Arduino/hardware//, keeping the tools as is
Move the tools folder to Arduino/hardware//tools, keeping the variant as is
Move both folders out of .arduino15/packages, to Arduino/hardware

Folder structures I am testing looks as follows:

Arduino/hardware//
Arduino/hardware//tools

Conclusion from my tests:

There are 4 possible configurations of moving variant and tools folders in/out .arduino15/. But, no matter what combination I use, the compiler.path and tools.path can resolve the correct paths. Arduino IDE can find the variant source code and compile just fine.
Only the default installation can do upload/program the board. Which, in my opinion, does not make any sense at all. Both compiler.path and tools.path use the same {runtime.tools.xxx.path} to resolve the correct path.

For my next experiment, I will make Arduino/hardware folder to have a similar structure like in .arduino15/

Will come back with the result if I can make it happen. I am pretty sure this is a bug in Arduino IDE.

EDIT: There is no other folder structure that works with Arduino/hardware. I don't remember where I got this configuration from, but I have been using it since Arduino 1.6.x. Very long time ago. I guess the Arduino team is pretty consistent keeping this structure intact until today.

Juraj · September 14, 2020, 11:36am

why do you want to move tools? let the STM32 boards package installation as it is and create your variant in hardware.

see the 'my_boards' repo. is is how I create my variants.

the referencing with "STM32:" will work for the STM32 package installed with Boards Manager

ekawahyu · September 14, 2020, 4:45pm

@Juraj, no I don't intend to move the tools if it works as is. But at some point I will if I need to.

I will look at your solution, but it is written a lot different that what I can see in STM32 core's board.txt. But, so far, when I tried to add it to my board.txt, it does not seem to fix the issue.

Juraj · September 14, 2020, 4:56pm

ekawahyu:
@Juraj, no I don't intend to move the tools if it works as is. But at some point I will if I need to.

I will look at your solution, but it is written a lot different that what I can see in STM32 core's board.txt. But, so far, when I tried to add it to my board.txt, it does not seem to fix the issue.

I created my my_boards variant for BluePill to not to have to select the options which are selectable with menus in core's boards.txt.
so what you see in my boards.txt are the same options but not a list, but always only one from the list

Topic		Replies	Views
Directory tree structure for hardware in Arduino/hardware ? IDE 1.x	4	1485	July 6, 2021
Folder/Directory layout for multiple platforms IDE 1.x	5	1735	May 5, 2021
Arduino IDE 1.5.x and 1.6.x hardware specification question Libraries	1	1025	May 6, 2021
Arduino IDE hardware directory. (solved) IDE 1.x	7	7031	May 5, 2021
Platform Specification for custom board similar to Arduino Zero Hardware Development	2	388	July 1, 2023

Platform Specification Question - is it a bug in Arduino IDE?

Related topics