What could prevent platform.local.txt from being read?

Development Tools IDE 1.x
1

I've got the Arduino IDE 1.8.12 running on an x86-64 Linux system (Debian "stable")., and I'm using it to compile a test project for a 649 ("Butterfly") board (it's actually upcycled from a scrap washing machine). I've installed appropriate board support, and the configurable settings appear in the IDE's menu. Everything works except that I've had to tweak AvrDude to work with the board's serial connection.

The progress messages show me the directory from which platform.txt is being loaded, and I've confirmed this by adding an extra compiler -D option:

Using core 'butterflycore' from platform in folder: /usr/local/src/butterflycore/ButterflyCore-master/avr
Detecting libraries used...
/home/markMLl/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -DPLATFORM_TXT=/usr/local/src/butterflycore/ButterflyCore-master/avr/platform.txt ...

However even if I edit the platform.local.txt file in the same directory to read like this

# These can be overridden in platform.local.txt. Be careful to not lose any
# platform-specific options already in the platform.txt file.

compiler.c.extra_flags=-DPLATFORM_LOCAL_TXT=/usr/local/src/butterflycore/ButterflyCore-master/avr/platform.txt
compiler.c.elf.extra_flags=-DPLATFORM_LOCAL_TXT=/usr/local/src/butterflycore/ButterflyCore-master/avr/platform.txt
compiler.S.extra_flags=
compiler.cpp.extra_flags=-DPLATFORM_LOCAL_TXT=/usr/local/src/butterflycore/ButterflyCore-master/avr/platform.txt
compiler.ar.extra_flags=
compiler.objcopy.eep.extra_flags=
compiler.elf2hex.extra_flags=

I can see absolutely no indication that it's being read.

This appears to work properly with "official" Arduino targets, since I use it regularly to make the name of the board type available to compiled code.

Can anybody suggest what obvious factor I'm overlooking?

MarkMLl

2

How did you get the IDE to recognize the boards platform at /usr/local/src/butterflycore/ButterflyCore-master/avr?

Usually it will only recognize the platforms at ~/Arduino/hardware/ButterflyCore-master/avr/ (where ~/Arduino is the sketchbook folder) or under ~/.arduino15/packages/ (which is the location they are installed when you use Boards Manager).

3

~/Arduino/hardware/ButterflyCore is a symlink to /usr/local/src/butterflycore/ButterflyCore-master which is where I was working on the bootloader (which needed a couple of tweaks). I decided it was safer to use a symlink than have multiple copies, because one /always/ ends up editing the wrong file :slight_smile:

I've tried running the IDE under strace but couldn't see anything useful, but I was looking for platform rather than going through call-by-call.

MarkMLl

4

You might give it a try without the symlink, just in case there's a problem with that usage. I use symlinks for the purposes of installing boards platforms and Arduino libraries constantly, but I've never tried it with platform.local.txt.

MarkMLl:
I've tried running the IDE under strace but couldn't see anything useful, but I was looking for platform rather than going through call-by-call.

Unfortunately, there is no consistently used term for this thing. It's actually more common for it to be referred to as "core" than "platform". It is also sometimes referred to as "package".

5

why would platform.local.txt work in the sketchbook's hardware folder? it is your folder, not one managed by IDE.
modify platform.txt

6

pert:
You might give it a try without the symlink, just in case there's a problem with that usage. I use symlinks for the purposes of installing boards platforms and Arduino libraries constantly, but I've never tried it with platform.local.txt.

I'd not have thought that was very likely, since the IDE is explicitly resolving the path to be in /usr/local for the sake of the output message. However I'm /not/ saying I don't think it's worth trying... unfortunately it doesn't appear to help. If I make up a real directory and do a verify I get

Using board '649' from platform in folder: /home/markMLl/Arduino/hardware/ButterflyCore/avr

Using core 'butterflycore' from platform in folder: /home/markMLl/Arduino/hardware/ButterflyCore/avr

Detecting libraries used...

/home/markMLl/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics

-DPLATFORM_TXT=/usr/local/src/butterflycore/ButterflyCore-master/avr/platform.txt
...

The value of the -D was what I'd hardcoded into the file, which is obviously now not quite right. the important thing is that I can verify that there's a platform.local.txt in the same place as the platform.txt, and it's still being ignored :frowning:

Unfortunately, there is no consistently used term for this thing. It's actually more common for it to be referred to as "core" than "platform". It is also sometimes referred to as "package".

I meant I was expecting to see it in a filename. I think that the problem is that the original arduino binary runs ...bin/java as a child, and I wasn't tracing into children. Here's what it looks like with strace's -f option, grepping for platform.local.txt:

18600 stat("/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt",  <unfinished ...>
18600 <... stat resumed> {st_mode=S_IFREG|0644, st_size=524, ...}) = 0
18600 access("/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", R_OK) = 0
18600 openat(AT_FDCWD, "/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", O_RDONLY) = 95
18600 stat("/home/markMLl/.arduino15/packages/arduino/hardware/avr/1.8.3/platform.local.txt", 0x7fb6cb5ec430) = -1 ENOENT (No such file or directory)
18600 stat("/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=524, ...}) = 0
18600 access("/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", R_OK) = 0
18600 openat(AT_FDCWD, "/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", O_RDONLY) = 95
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 access("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", R_OK) = 0
18600 openat(AT_FDCWD, "/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", O_RDONLY) = 95
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt~", {st_mode=S_IFREG|0644, st_size=788, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt~", {st_mode=S_IFREG|0644, st_size=788, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt~", {st_mode=S_IFREG|0644, st_size=788, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt",  <unfinished ...>
18600 <... stat resumed> {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt",  <unfinished ...>
18600 <... stat resumed> {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt",  <unfinished ...>
18600 <... stat resumed> {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt~", {st_mode=S_IFREG|0644, st_size=788, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt~", {st_mode=S_IFREG|0644, st_size=788, ...}) = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt~", {st_mode=S_IFREG|0644, st_size=788, ...}) = 0
18664 newfstatat(AT_FDCWD, "/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=524, ...}, 0) = 0
18664 openat(AT_FDCWD, "/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", O_RDONLY|O_CLOEXEC) = 3
18664 newfstatat(AT_FDCWD, "/home/markMLl/.arduino15/packages/arduino/hardware/avr/1.8.3/platform.local.txt", 0xc0002cced8, 0) = -1 ENOENT (No such file or directory)
18664 newfstatat(AT_FDCWD, "/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=593, ...}, 0) = 0
18664 openat(AT_FDCWD, "/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", O_RDONLY|O_CLOEXEC) = 3
18675 newfstatat(AT_FDCWD, "/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=524, ...}, 0) = 0
18675 openat(AT_FDCWD, "/usr/local/lib/arduino/arduino-1.8.12/hardware/arduino/avr/platform.local.txt", O_RDONLY|O_CLOEXEC) = 3
18675 newfstatat(AT_FDCWD, "/home/markMLl/.arduino15/packages/arduino/hardware/avr/1.8.3/platform.local.txt",  <unfinished ...>
18675 <... newfstatat resumed> 0xc0002c8d38, 0) = -1 ENOENT (No such file or directory)
18675 newfstatat(AT_FDCWD, "/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=593, ...}, 0) = 0
18675 openat(AT_FDCWD, "/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", O_RDONLY|O_CLOEXEC <unfinished ...>
18675 <... openat resumed> )            = 3

Also I can explicitly see this equence:

18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.txt", {st_mode=S_IFREG|0644, st_size=7364, ...}) = 0
18600 access("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.txt", R_OK) = 0
18600 openat(AT_FDCWD, "/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.txt", O_RDONLY) = 95
18600 fstat(95, {st_mode=S_IFREG|0644, st_size=7364, ...}) = 0
18616 futex(0x7fb6bc32e57c, FUTEX_WAIT_PRIVATE, 0, {tv_sec=4, tv_nsec=999999651} <unfinished ...>
18600 read(95, "\n# Arduino AVR Core and platform"..., 8192) = 7364
18600 fstat(95, {st_mode=S_IFREG|0644, st_size=7364, ...}) = 0
18600 lseek(95, 0, SEEK_CUR)            = 7364
18600 read(95, "", 8192)                = 0
18600 close(95)                         = 0
18600 stat("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 access("/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", R_OK) = 0
18600 openat(AT_FDCWD, "/home/markMLl/Arduino/hardware/ButterflyCore/avr/platform.local.txt", O_RDONLY) = 95
18600 fstat(95, {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 read(95, "\n# These can be overridden in pl"..., 8192) = 593
18600 fstat(95, {st_mode=S_IFREG|0644, st_size=593, ...}) = 0
18600 lseek(95, 0, SEEK_CUR)            = 593
18600 read(95, "", 8192)                = 0
18600 close(95)                         = 0

So the file is apparently being found and read, but the content ignored.

I think this is getting outside what a non-IDE-developer can usefully comment on. However I'd note that most if not all of the platform.local.txt files on this system already contain board identification text which should be easily recognisable in the compiler output, and since it's not in fact there it suggests that it's not a question of an incorrectly-read file overriding the right one.

MarkMLl

7

Please post your current platform.txt and platform.local.txt

8

platform.local.txt works just fine for me in platforms installed to the sketchbook. The Arduino IDE handles configuration files exactly the same regardless of where the platform is installed.

If platform.local.txt is not working in sketchbook installed platforms, that's a bug.

9

pert:
Please post your current platform.txt and platform.local.txt

Attached, from /home/markMLl/Arduino/hardware/ButterflyCore/avr

The platform.txt file is unmodified from the ButterflyCore project on Github, except for the extra -D as a marker so that I could see that it was being read. The platform.local.txt has extra defines in it to allow the board type to be incorporated into a sketch being built, they work correctly with the pukka Arduinos I've tested (Uno, Mega2560, possibly others) and I think with some others (e.g. "Blue Pill" but I'd have to check... major thing is they're OK with the Arduino IDE for other target platforms).

MarkMLl

platform.local.txt (593 Bytes)

platform.txt (7.19 KB)

10

the ButterflyCore has the 'extras' set in boards.txt

11

Juraj gave the correct answer while I was writing this, but it has some additional information so I'll post it anyway:

OK, here's the problem:

# Compiler link time optimization
649.menu.LTO.Os=Disabled (default)
649.menu.LTO.Os.compiler.c.extra_flags=
649.menu.LTO.Os.compiler.c.elf.extra_flags=
649.menu.LTO.Os.compiler.cpp.extra_flags=
649.menu.LTO.Os.ltoarcmd=avr-ar

649.menu.LTO.Os_flto=Enabled
649.menu.LTO.Os_flto.compiler.c.extra_flags=-Wextra -flto
649.menu.LTO.Os_flto.compiler.c.elf.extra_flags=-w -flto
649.menu.LTO.Os_flto.compiler.cpp.extra_flags=-Wextra -flto
649.menu.LTO.Os_flto.ltoarcmd=avr-gcc-ar

You can see that, despite the promise made in the comment in platform.txt (which is a result of copy/pasting from another platform), the author of the ButterflyCore boards platform used the "extra_flags" properties in boards.txt. The definitions in that file have priority over the definitions in platform.local.txt. So the definition in platform.local.txt overrides the definition in platform.txt, but the definition in boards.txt overrides the definition in platform.local.txt.

Best practices for platform developers would be to leave those "extra_flags" properties free for the user. I actually have made a proposal about how Arduino can help to avoid this very situation:

I also have made a proposal to document the override priorities of the various places you can set build properties.

These "extra_flags" properties are only arbitrary properties without any special handling from the build system, so you are free to rename them and create as many of your own as you like. The easiest fix would be to use unique names for the properties you use in platform.local.txt and then add those properties into the appropriate compilation recipes.

For example, you could do this in platform.local.txt:

compiler.cpp.user_extra_flags=-DPLATFORM_LOCAL_TXT=/usr/local/src/butterflycore/ButterflyCore-master/avr/platform.txt

then add that custom property ({compiler.cpp.user_extra_flags}) to the definition of recipe.cpp.o.pattern in platform.txt:

## Compile c++ files
recipe.cpp.o.pattern="{compiler.path}{compiler.cpp.cmd}" {compiler.cpp.flags} -mmcu={build.mcu} -DF_CPU={build.f_cpu} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} {compiler.cpp.extra_flags} {compiler.cpp.user_extra_flags} {build.extra_flags} {includes} "{source_file}" -o "{object_file}"
12

Aha! Thanks very much for that @Juraj, @Pert. I'm /so/ glad it turned out to be something so simple, and will tinker as suggested.

"When a programmer has a hard time finding a bug, it's because he's looking in the wrong place." I should obviously have grepped for any extra occurrences of the lines in platform.local.txt.

The indication in the IDE output showing where the files are being looked for is particularly useful, but possibly a note every time one of the platform.board/programmer files is read would help for things like this. Still wouldn't help though if somebody (like the maintainer of ButterflyCore) did something unexpected and arguably naughty.

Slightly later:

For info, I've modified my platform.txt to have a brief warning that those settings can also be overridden in boards.txt, and my platform.local.txt to read

# These can be overridden in platform.local.txt, and potentially in boards.txt. Be
# careful to not lose any platform-specific options already in the platform.txt file.

compiler.c.extra_flags=
compiler.c.elf.extra_flags=
compiler.S.extra_flags=
compiler.cpp.extra_flags=
compiler.ar.extra_flags=
compiler.objcopy.eep.extra_flags=
compiler.elf2hex.extra_flags=

# ButterflyCore overrides the above flags. The last-ditch fix below requires that the
# compile patterns ("recipes") in platform.txt be modified.

compiler.c.boardname_extra_flags=-DARDUINO_BOARD_NAME={build.board} "-DARDUINO_BOARD_TEXT={name}"
compiler.c.elf.boardname_extra_flags=-DARDUINO_BOARD_NAME={build.board} "-DARDUINO_BOARD_TEXT={name}"
compiler.cpp.boardname_extra_flags=-DARDUINO_BOARD_NAME={build.board} "-DARDUINO_BOARD_TEXT={name}"

Hence the build output shows

... -DARDUINO_ARCH_AVR -DARDUINO_BOARD_NAME=AVR_ATmega649 -DARDUINO_BOARD_TEXT=ATmega649 -I/usr/local/src/butter...

and a test script can output

Project: /usr/local/src/arduino-examples/double-tap-and-hello/double-tap-and-hello.ino Sep 19 2020 09:20:18
Target: AVR_ATmega649 (ATmega649) 4.00 MHz v6.2 Reset: 0x08
Hello, World! LED is on pin 13.

QED, so to speak :slight_smile:

MarkMLl

13

There's an interesting suggestion in a reply to my proposal:

We could also print warnings to the console at compile time whenever something is overridden, to help users troubleshoot.

The biggest consideration with this is that overrides are a normal and expected part of the way the Arduino platform build system works. Every time you use one of those "extra_flags" properties, you're overriding the empty definition in platform.txt (which are there to make sure the properties are always defined). So this sort of output would be "noisy" and it's only of interesting to platform developers. For that reason, there would need to be a mechanism to enable this output.

14

pert:
The biggest consideration with this is that overrides are a normal and expected part of the way the Arduino platform build system works. Every time you use one of those "extra_flags" properties, you're overriding the empty definition in platform.txt (which are there to make sure the properties are always defined). So this sort of output would be "noisy" and it's only of interesting to platform developers. For that reason, there would need to be a mechanism to enable this output.

Agreed. One possibility would be to take this opportunity to introduce --help --version and --verbose options (and their Windows equivalents). However I imagine that would be beyond many less-experienced users... I'd suggest a "verbosity" setting in the preferences.txt file since the IDE (to its great credit- wish more developers did this) tells the user exactly where it is.

I need to find time to work out how to use Github to upload my modified Optiboot for the 649. The reset circuitry is decidedly manky on that board, and fixing that took it over 256 bytes so I had to find a way of saving a few bytes. Job for today though is getting Avrdude to handle either sense of DTR... the code's already in there but not for the stk500 protocol.

You've probably seen Surgery On The Arduino IDE Makes Bigger Serial Buffers | Hackaday which discusses the flexibility of the IDE.

MarkMLl

15

MarkMLl:
One possibility would be to take this opportunity to introduce --help --version and --verbose options

The Arduino IDE's CLI does have --verbose and --version options. However, no --help, which is certainly a glaring deficiency.

Arduino CLI has very nice command line help though.

I'd suggest a "verbosity" setting in the preferences.txt file since the IDE (to its great credit- wish more developers did this) tells the user exactly where it is.

I agree that would be the way to go. I'd never want to expose normal users to a "report build property overrides" option in the GUI. This is something that will be useful to 0.001% of Arduino IDE users at most. However, we can feel more free to add advanced options in preferences.txt, since the users who would get confused by seeing them in the GUI will never look in preferences.txt.

You can also control the properties in preferences.txt (or any of the other configuration files) via the --pref option.

I need to find time to work out how to use Github to upload my modified Optiboot for the 649.

I'm happy if you'll make the effort to share your work! You might consider trying to push the fixes back upstream. I know westfw has been pretty open to making optiboot compatible with all AVR. The number of supported parts has really jumped up in the last couple years!

MarkMLl:
You've probably seen Surgery On The Arduino IDE Makes Bigger Serial Buffers | Hackaday which discusses the flexibility of the IDE.

Yeah, I am glad to see more awareness of the Arduino platform system and its documentation, but there are some serious misconceptions and errors in that article. It seems like it usually happens when "experts" who don't like and don't use Arduino talk about it.

I actually created a boards platform in response to that article to demonstrate the "right way" to do it:

Instead of editing the MightyCore platform itself, you can create an "extension" platform that references all its resources from MightyCore, allowing it to be very minimal. I was under a 45 minute time limit to accomplish the task. I think I could have gotten it more minimal if I had a little more time.

This "extension platform" means that you don't need to make the same changes over and over again on every release of MightyCore. This modification to MightyCore is something the platform's author, MCUdude has explicitely said they will not add to MightyCore, so it's not something that can be pushed upstream. The more minimal approach would have been to just do a boards.local.txt, but that still requires the users to find the MightyCore installation and add this file after every update. Conversely, you could easily provide a platform index file to make it super easy for anyone to install and update the extension platform via the Arduino Boards Manager.

16

pert:
Yeah, I am glad to see more awareness of the Arduino platform system and its documentation, but there are some serious misconceptions and errors in that article. It seems like it usually happens when "experts" who don't like and don't use Arduino talk about it.

Particularly if they are self-annointed Eclipse experts. Their fervour matches that of the 4DOS users from years ago: every question results in "stop using your existing tools and use Eclipse". Which is extremely unfortunate, because whatever criticism one levels at the Arduino IDE's editor UI it really is groundbreaking when one considers its boards/libraries management etc.

I actually created a boards platform in response to that article to demonstrate the "right way" to do it:
GitHub - arduino/ArduinoCore-Z80-MBC
Instead of editing the MightyCore platform itself, you can create an "extension" platform that references all its resources from MightyCore, allowing it to be very minimal. I was under a 45 minute time limit to accomplish the task. I think I could have gotten it more minimal if I had a little more time.

This "extension platform" means that you don't need to make the same changes over and over again on every release of MightyCore. This modification to MightyCore is something the platform's author, MCUdude has explicitely said they will not add to MightyCore, so it's not something that can be pushed upstream. The more minimal approach would have been to just do a boards.local.txt, but that still requires the users to find the MightyCore installation and add this file after every update. Conversely, you could easily provide a platform index file to make it super easy for anyone to install and update the extension platform via the Arduino Boards Manager.

OTOH there are quite possibly people now using or at least aware of the Arduino IDE who got here because of that and related articles. I came across it after I'd done my initial board naming hack, I found it vaguely useful when getting the IDE to pick up the ButteflyCore stuff. I've certainly considered hacking together an improved editor (I use https://www.lazarus-ide.org/ which would make that an easy job) but it's all the stuff that goes on behind the scenes. I bet that somebody, somewhere, has said to himself "now if we could just redo that using Prolog..." :slight_smile:

In any event, from the POV of the sort of problem we're discussing having the IDE (a) announce the location of preferences.txt and (b) announce where it's looking for platform.txt etc. puts it ahead of most of the competition, even if it's not delivering a blow by blow account of what it's doing.

Regards,

MarkMLl