Reference for .ino file structure? Beyond setup() and loop()

[ptillisch]

@westfw

Someone should point out that modifying the compile/etc commands is NOT “the arduino way”

Yes, it is important to understand that the --build-property flag is provided to allow advanced users to do very advanced things when using the Arduino CLI tool, which is specifically designed to support advanced users.

@westfw

is not commonly done to achieve the sort of results that you are describing. Better to use a config.h that your sketch includes.

Yes, and in fact the concerns about developers making this sort of thing an interface for standard users of their projects has historically motivated the Arduino company to decline requests for making it easier for users to define global macros.

At least when it comes to a user interface (and even generally), the C preprocessor is something that should only be used as a last resort when there is absolutely no other alternative. Projects that overuse the preprocessor are absolute nightmares for everyone involved.

@westfw

I’m not sure the builder will get dependencies right if all you change is a compile flag.

I'm not sure I understand what you mean by this. Would you mind providing a more detailed explanation?

Maybe you are concerned that macros defined via the --build-property flag will not have the expected effect on code for which compiled objects were cached during a previous compilation? If so, this is not a problem. The sketch build system is smart enough to do a recompile if the platform properties have been changed via the --build-property flag:

internal/arduino/builder/build_options_manager.go

@drankinatty

Hopefully the github issue to add an easy way to do this will get some movement and provide a solution that doesn't have potential unwanted side effects. It's been open for a while, so maybe it's nearing the point of implementation.

Please note that there is nothing to implement in regards to the build system. The only thing that needs to be done is to formally state in the Arduino Platform Specification that platforms should provide a standardized set of properties dedicated exclusively for users to inject arguments into the compilation commands.

The platform framework already fully supports this. The only problem is that the established convention of providing such properties has never been formally documented.

[westfw]

@drankinatty

I long ago learned that if you understand how your code is actually built, you can then tell and IDE how you want it done instead of just hoping the IDE config gets it right.

While I don't disagree "in principle", understand that the Arduino environment was created (to a significant extent) to hide the build process from users.

@drankinatty

if using, e.g. compile.c.extra_flags isn't what one is supposed to do, then what are those build properties for?

They're primarily for the creators of new platforms, that can potentially have boards or board options that require extra options. If you look at one of the more complex 3rd party platform packages, like MiniCore, there is an IDE "Tools" option for specifying whether LTO (link time optimization) is enabled or disabled, and that's passed to the actual build tools via c.extra_flags and cpp.extra_flags
I think that the boards.txt/platform.txt format was established before the arduino-builder application, so essentially all of those variables are for "internal use by the IDE (oh, I guess you can specify them on the command line if you really want.)"

There is also apparently build.extra_flags (at least, in boards.txt and platform.txt; I only assume that it'd be allowed via arduino-builder) that seems closer to what you want. Except that it's regularly used to pass USB options (Product ID) to native USB boards.

AFAIK, there's no "extra" flags variable explicitly reserved (or appendable) for command-line use.

(Hmm. For @ptillisch Is arduino-builder essentially the same tool as the arduino-cli? github says it's phased out in-favor-of. I thought that the new IDE uses the cli tools internally, but I don't see any invocations of arduino-cli in the verbose IDE output (are they hidden?))

[ptillisch]

@westfw

Is arduino-builder essentially the same tool as the arduino-cli?

The Arduino CLI codebase started from the arduino-builder codebase. arduino-builder was then changed to be essentially just a wrapper around Arduino CLI (Arduino CLI could not be used as a drop-in replacement for arduino-builder due to having a different command line interface).

However, the arduino-builder project has been completely abandoned for years, while a tremendous amount of development work has been done on Arduino CLI, so Arduino CLI is now significantly advanced beyond the capabilities of arduino-builder.

@westfw

I thought that the new IDE uses the cli tools internally

That is correct. Arduino CLI provides most of the non-GUI capabilities for Arduino IDE 2.x under the hood.

@westfw

I don't see any invocations of arduino-cli in the verbose IDE output (are they hidden?)

They are not hidden. Other than a single arduino-cli daemon invocation on each startup, and an arduino-cli config invocation on the first run to generate the configuration file, Arduino IDE doesn't use the command line interface of Arduino CLI. It instead uses the gRPC interface:

https://arduino.github.io/arduino-cli/latest/integration-options/#the-second-pillar-grpc-interface

So all the communication between Arduino IDE and Arduino CLI is occurring through localhost. This approach has significant advantages compared to using the command line interface, but unfortunately it does mean that it is not possible to expose the communication between Arduino IDE and Arduino CLI in the Output panel, as you would see done for arduino-builder when using Arduino IDE 1.x.

If you do want to see the communication, you can enable the arduino.cli.daemon.debug option in Arduino IDE's advanced settings. This will cause it to be included in the log file that is generated by Arduino IDE. It will also be logged to the terminal if you started Arduino IDE from the terminal. However, I don't recommend that because this setting results in a massive amount of log output which might exceed the terminal's throughput.

[drankinatty]

@westfw

the Arduino environment was created (to a significant extent) to hide the build process from users.

Well, you all have done a very good job with it. Not the hiding, but with the build environment in general. I'm just so glad I found arduino-cli to begin working with arduino with. It would have been simply painful to start with the IDE and then have to pick apart the config and file structure to accomplish the same learning the cli provides in a day or two :)

Though don't let that take away from the IDE for those just beginning dabbling in programming and microcontrollers. That greatly simplifies things compared to the TI RM46 I departed x86 and waded into the microcontroller world with. CMSIS and bit flipping at the register level is a painful introduction as well.

But don't "phase out" the cli. It's worth it's weight in gold for more seasoned users that come with a command line understanding. It may have it's quirks and limitations, but it is still much appreciated verses pointing-clicking and praying with dialogs in an IDE not being sure what's taking place behind the scene.

Double thanks to all, I'm basically up and running, both little nanos plugging along, all systems tested, and no problems (other than the normal veil of confusion over a new platform/build system that slowly fades). I'm just glad my son left me at least one good bootloader between the two nanos to start from :)

[ptillisch]

@westfw

They're primarily for the creators of new platforms, that can potentially have boards or board options that require extra options. If you look at one of the more complex 3rd party platform packages, like MiniCore, there is an IDE "Tools" option for specifying whether LTO (link time optimization) is enabled or disabled, and that's passed to the actual build tools via c.extra_flags and cpp.extra_flags

This is incorrect. They were explicitly intended to allow the user to inject arguments into the compilation commands:

Add (empty) compiler.*.extra_flags variables in platform.txt

There is absolutely no need for MiniCore to use these properties. The platform developer is free to define any number of arbitrary properties. Arbitrary properties would be functionally identical to the compiler.c.extra_flags and compiler.cpp.extra_flags in this use case (in fact compiler.c.extra_flags and compiler.cpp.extra_flags are just arbitrary properties).

The case could be made for using these properties in a minimal platform that references resources from another platform:

https://arduino.github.io/arduino-cli/latest/platform-specification/#referencing-another-core-variant-or-tool

since in that case it is not necessary for the board platform (the referencing platform) to define compilation command patterns, and so it would be convenient to use these properties in order to inject arguments into the compilation command generated by the referenced platform's patterns. However, the board platform is free to define its own patterns so the effort required to copy/paste the pattern from the referenced platform and add a reference for the arbitrary property defined in the board definition is utterly trivial. The pattern is not terribly complex, and also not prone to frequent changes, so the maintenance burden of that added code is negligible.

And note that MiniCore does not reference another platform, so that argument is not at all applicable to MiniCore anyway.

---

The reason we see foolish things like this done in Arduino platforms is because too often the platform developers never take the time to actually understand the platform framework and instead cursorily copy/paste their way through the platform configuration. That was understandable in the early days when Arduino didn't provide documentation for the framework, but there has been pretty good documentation for quite a few years now:

History for Arduino IDE 1.5 3rd party Hardware specification ·...

History for Arduino IDE 1.5 3rd party Hardware specification ·...

History for docs/platform-specification.md - arduino/arduino-cli

The thing I think we still don't do a good job of communicating is about the two distinct usage classes of properties, as I explain here:

Technical question regarding `includes.txt`

Hi @andriandreo. The term we use for the things like xiaonRF52840.build.includes is "properties" (or in cases where that general term might be ambiguous "platform properties"). We can define a property like this: salutation=Hello And reference a property in the definition of another property: message={salutation} world! An important thing to understand about properties is that they fall into multiple distinct classes of usage: Framework-Defined Properties Some properties (e.g., runtime.p…

So platform developers might be led to think that there is some magical internal usage of compiler.c.extra_flags and compiler.cpp.extra_flags by the framework that means they should be used instead of doing the responsible thing and defining new arbitrary properties for internal use in their platform.

@westfw

I think that the boards.txt/platform.txt format was established before the arduino-builder application

boards.txt goes way back to 2007's version "0010":

Initial boards file.

platform.txt is more recent being introduced in 2012's version 1.5:

Added platforms.txt, and update boards.txt with platform=avr

but that was still long before the advent of arduino-builder circa 2015.

[westfw]

@ptillisch

This is incorrect. They were explicitly intended to allow the user to inject arguments into the compilation commands:

Ah. I see:

# This can be overridden in boards.txt                                                                                                                                        
build.extra_flags=                                                                                                                                                            
                                                                                                                                                                              
# These can be overridden in platform.local.txt                                                                                                                               
compiler.c.extra_flags=                                                                                                                                                       
compiler.c.elf.extra_flags=                                                                                                                                                   
compiler.S.extra_flags=                                                                                                                                                       
compiler.cpp.extra_flags=                                                                                                                                                     
compiler.ar.extra_flags=                                                                                                                                                      
compiler.objcopy.eep.extra_flags=                                                                                                                                             
compiler.elf2hex.extra_flags=    

"local" would apply to extra arduino-cli flags as well, and build.extra_flags would have been more appropriate. Or as you say, entirely new flags particular to that platform. (arduino-cli automagically supports defining any argument that's defined in platform.txt or boards.txt, right?)

I see that there is no set of "extra" flags applied to all build commands (similar to build.extra_flags): would it then be a simple matter of defining a new (say) local.build.extra_flags in platform.txt, and adding it to the recipes (also in that platform.txt)?

[ptillisch]

@westfw

"local" would apply to extra arduino-cli flags as well

Yeah, at the time those properties were added to the platform, platform.local.txt was probably the most prominent mechanism for users to set platform properties (it was also possible to do via the --pref flag of the Arduino IDE 1.x command line, but I think that mechanism was primarily intended to be used for setting standard IDE preferences ala preferences.txt). So I think this was just the way it made sense to describe the user injecting arguments into the compilation commands at that time.

@westfw

arduino-cli automagically supports defining any argument that's defined in platform.txt or boards.txt, right?

Yes, at least in terms of compilation, there aren't any restrictions on what can be done with the --build-property flag. You can even define properties that aren't defined in those files.

@westfw

would it then be a simple matter of defining a new (say) local.build.extra_flags in platform.txt, and adding it to the recipes (also in that platform.txt)?

Correct. I think it would probably be a good thing to add to the standardized set of properties dedicated to use by the user. It is great that per-pattern granularity was provided, but it might be that a user wants to inject certain arguments universally and they would find it inconvenient to have to set multiple properties in that case.