Easiest way to configure arduino-cli for UNO and NANO

I have this project to program Arduino boards with using logic gates n such. It is ment for certain hobbyist who cannot program at al.

It is written in processing and what it does, is generating source code to be used with the arduino IDE.

My manual points to another manual to install and use the arduino-IDE. But the end-users still have to download and install the arduino IDE.

It transforms something like this in a functional .ino file. In the IDE you just have to select com port, board and hit upload.

I would love if I can use arduino-CLI instead of the IDE. I have one small problem with that. Sure I can copy an arduino-cli.exe and some supporting files in a zip file, but that won't work because one would need certain configurations. You need to install the AVR cores for example.

The processing program itself comes as an executable with embedded java. This allows it to work out of the box on windows (also got linux binaries) computers without having to run an installer or something. It suffices to just unzip the zip.

But Arduino-cli uses that absurd 'hidden' appdata folder. And I don't know if or how I can move this.

So TL:DR part: I want my processing program to call batch files which in turn run a fully configured ( <= can do UNO and NANO boards) arduino-cli.exe to upload the generated INO file to an Arduino board.

What would the best method be for me to deploy and configure arduino-cli for my project. So that the end users (who can barely operate a windows PC) experience little to no problems with installation / unzipping.

I suppose I could make an install batch file which configures arduino.cli for you?
Or should I really learn to make an installer or something?
Can I get all the needed pre-configurated files in the same folder as the cli.exe? (instead of that appdata folder)

I really not know what is wise to do.

Pointers and tips of any kind would be appreciated :+1:

:tumbler_glass:
Bas

Look into the "portable" IDE install option. I moves all of the things that would normally be in Appdata (as well as the sketchbook) into your choice of folder.

Although C:\Users\<username>\AppData\Local\Arduino15\ is the default location on Windows, it can be configured via the directories.data configuration key. There is documentation for configuring Arduino CLI here:

https://arduino.github.io/arduino-cli/latest/configuration/

That page mentions it in passing, but worth also noting here that there is an arduino-cli config command for working with the configuration.

I think it should be possible for you to accomplish your goal using one of the configuration methods. Give it a try. If you get stuck, let us know and we'll see if we can find a solution.

Thank you for your answers. I have another question.

When I call arduino-cli.exe from me script

x = os.system('arduino-cli compile -b arduino:avr:nano -e')

I don't specify any directory related to arduino-cli's configuration files

In the program files folder where the binary lies. There are no supporting files either besides that licence.

So when I just call arduino-cli .exe how does 'it' know where to look for it's configuration files?

Is the path to the appdata folder somehow in the binary or something??

Meanwhile I'll try to use the config option and see what happens. I was thinking to put both my own program as well as all the compile tooling on USB stick to see if I can run it on other 'virgin' computers.

Regards,

Bas

It uses the default directories.data location.

Arduino CLI queries the operating system for the path of the folder.

I had some troubleties setting the path, but eventually with the --help flag and some trial and error I managed to change the folder.

arduino-cli.exe config set directories.data "C:\Program Files (x86)\Arduino-CLI\configFiles"

With the first build, arduin-cli.exe filled it for me with this:


And ofcourse the expected error that I need to install the AVR cores.

So I replaced the content of this folder by the content of the old folder and I can compile and upload again.

This leaves me with 2 questions.

One of the files which was not generated is the arduino-cli.yaml file, where is this one good for?

Arduino CLI queries the operating system for the path of the folder.

As I understand, I essentially told my OS that arduino-cli wants to use a different path with the arduino-cli config command?

If understood correctly:
If I would ship a zip file with the binary as well the config files it still would not work as the end-user would still have to tell his OS (by means of arduino-cli config .. ) where the directories.data is, right?

So final question (I hope :stuck_out_tongue: ). Do I need to make a setup.bat file which does something like:
arduino-cli.exe config set directories.data "current_path_of_all_my_stuff\configFiles"
And ofcourse: would that suffice?

Regards :coffee:

Bas

Great job!

I'm not sure I understood correctly what you mean by this. Please add a forum reply here that provides a more detailed description to help me to understand it.

No, you told Arduino CLI that it should use a different path. The operating system only tells Arduino CLI the base path for the default directories.data location.

That is one way to do it.

However, my recommendation would be for you to do it in the same script you are using to invoke the arduino-cli compile command. This way your project could be used by Linux and macOS users as well instead of being Windows-specific.

I also just avoid using Windows batch files as much as possible as a principle. They are fine when you only need to run some commands, but once you try to do anything more advanced they are such a nightmare. Even if you start with a simple batch file, the complexity of a system tends to increase over time and before you know it you will be tearing your hair out trying to figure out how to use that terrible language to do things that would utterly trivial in Python.

That seems legit. Going to do some more --help command line entries and RTF(fine)M

About that yaml. In my appdata folder I happened to notice this .yaml file
afbeelding
Inside it there are is some config information

board_manager:
  additional_urls:
  - https://arduino.esp8266.com/stable/package_esp8266com_index.json
  - http://drazzy.com/package_drazzy.com_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  data: C:\Program Files (x86)\Arduino-CLI\configFiles
  downloads: C:\Users\sknippels\AppData\Local\Arduino15\staging
  user: C:\Users\sknippels\Documents\Arduino

When I changed the path and ran build n upload. Arduino-cli filled the new config folder with all kind of files but this particular file was not created. I think it maybe was once created by arduino-cli when installing extra boards or libraries or something. But I really have no clue when it was made, why and if it is important for something? I do sometimes program an attiny or esp.

No, you told Arduino CLI that it should use a different path

I understand but when I ran build after the path change, arduino-cli filled the new folder with the new config files n such. I thought that the OS told arduino-cli where this new path was.

I also just avoid using Windows batch files as much as possible as a principle.

I do, but you must understand that I deploy software to people who cannot do that much with a PC. Letting them install python on their system is therefor a big no-no. I ofcourse use python to build and upload arduino programs. I even use it to generate state machine code for me and to assemble new code projects.

I have for instance an open source project which involves PCB SMT assembly. With a guide people can directly order a fully assembled board at JLCPCB. But what they buy are boards with atmega's without bootloaders. So I needed a method to teach these people to flash that board using an arduino. Ofcourse I made manuals on howto install arduino IDE and do ISP programming.

But to make life easier, I made this folder. This folder contains all you need. It has avrdude + config stuff and it has 2 binaries. It works on all windows PCs and it need no extra tooling like Python or the arduino IDE.

With upload_ISP.bat you can flash any UNO with the ISP program and with the upload_OSSD.bat you can program the board via the UNO. It even sets the fuse bits correctly in the process (achieved by help from this forum :wink: )
afbeelding

Anyways. I go figure out howto set a parameter for the config path EDIT: I miss understood. I get that I should simply always call the config command prior to build and uploading.

.. and hopefully I get to test it in a week or two. It would be neat If I can turn my logic gate program into an actual IDE.

Kind regards :coffee:

Bas

That is the Arduino CLI configuration file.

There is a "chicken or the egg" situation with customizing the directories.data location. The documentation says:

https://arduino.github.io/arduino-cli/latest/configuration/#locations

Configuration files in the following locations are recognized by Arduino CLI:

[...]
2. Arduino CLI data directory (as configured by directories.data)

However, if a custom location for directories.data is configured via the configuration file, how can Arduino CLI know to use the configuration file at <directories.data>/arduino-cli.yaml without reading the configuration file.

So Arduino CLI still uses the file at C:\Users\<username>\AppData\Local\Arduino15\arduino-cli.yaml under these conditions even though C:\Users\<username>\AppData\Local\Arduino15 is not directories.data per the configuration file.

A useful technique for understanding which configuration file is in use is running the arduino-cli config dump command with the --verbose flag:

> ./arduino-cli config dump --verbose
INFO[0000] Using config file: C:\Users\per\AppData\Local\Arduino15\arduino-cli.yaml 
INFO[0000] arduino-cli.exe version 0.34.2
INFO[0000] Executing `arduino-cli config dump`
board_manager:
  additional_urls: []
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  data: e:\arduino-cli-directories\data
  downloads: C:\Users\per\AppData\Local\Arduino15\staging
  user: C:\Users\per\Documents\Arduino
library:
  enable_unsafe_install: false
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

If you want the configuration file from the actual custom directories.data folder to be used, you can configure it via the ARDUINO_DIRECTORIES_DATA environment variable:

https://arduino.github.io/arduino-cli/latest/configuration/#environment-variables

>set ARDUINO_DIRECTORIES_DATA=e:\arduino-cli-directories\data  

>arduino-cli config dump --verbose
INFO[0000] Using config file: e:\arduino-cli-directories\data\arduino-cli.yaml 
INFO[0000] arduino-cli version 0.34.2
INFO[0000] Executing `arduino-cli config dump`
board_manager:
  additional_urls: []
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  data: e:\arduino-cli-directories\data
  downloads: C:\Users\per\AppData\Local\Arduino15\staging
  user: C:\Users\per\Documents\Arduino
library:
  enable_unsafe_install: false
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

I was led to believe you are giving them a Python script to run by your previous comment:

My bad :wink:

I have some succes and a minor fail.

My folder setup:

This is my main processing folder. For testing purpose I added the folder with Arduino-CLI.exe as well as the subfolder with the config files.

Further there is the arduinoProgram sketch folder with the .ino file as generated by my processing program.

I devised this batch file

echo off

set ARDUINO_DIRECTORIES_DATA=%~dp0\Arduino-cli\configFiles
Arduino-cli\arduino-cli compile --verbose -b arduino:avr:uno .\arduinoProgram

pause

It seems to partially work

C:\Users\sknippels\Documents\hobbyProjects\functionBloX>echo off
FQBN: arduino:avr:uno
Using board 'uno' from platform in folder: C:\Users\sknippels\Documents\hobbyProjects\functionBloX\Arduino-cli\configFiles\packages\arduino\hardware\avr\1.8.6
Using core 'arduino' from platform in folder: C:\Users\sknippels\Documents\hobbyProjects\functionBloX\Arduino-cli\configFiles\packages\arduino\hardware\avr\1.8.6

This seems legit.

But I forgot a minor detail:

Detecting libraries used...
C:\Users\sknippels\Documents\hobbyProjects\functionBloX\Arduino-cli\configFiles\packages\arduino\tools\avr-gcc\7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR -IC:\Users\sknippels\Documents\hobbyProjects\functionBloX\Arduino-cli\configFiles\packages\arduino\hardware\avr\1.8.6\cores\arduino -IC:\Users\sknippels\Documents\hobbyProjects\functionBloX\Arduino-cli\configFiles\packages\arduino\hardware\avr\1.8.6\variants\standard C:\Users\sknippels\AppData\Local\Temp\arduino\sketches\0B368E004C8EA800747B3B565A371CFB\sketch\arduinoProgram.ino.cpp -o nul
Alternatives for Servo.h: []
ResolveLibrary(Servo.h)
  -> candidates: []
In file included from C:\Users\sknippels\Documents\hobbyProjects\functionBloX\arduinoProgram\arduinoProgram.ino:1:0:
C:\Users\sknippels\Documents\hobbyProjects\functionBloX\arduinoProgram\functionBlocks.h:2:10: fatal error: Servo.h: No such file or directory
 #include <Servo.h>
          ^~~~~~~~~
compilation terminated.

'It' cannot find Arduino's Servo library anymore.

I am not exactly sure but I think it is because I made a copy of Arduino-CLI.exe and put somewhere else which isn't in the Program Files folder.

I suppose I need to copy atleast the libraries folder. But do I need more stuff from this folder such as the drivers or something?

I tried by copying the libraries folder into the src/ directory of the arduino sketch and replace
#include <Servo.h>
By
#include "src/libraries/Servo/src/Servo.h"

But that seems to relocate the problem... obviously this isn't the approach. :man_facepalming:

...t_CircuitPlayground.cpp:30:10: fatal error: Adafruit_Circuit_Playground.h: No such file or directory
 #include <Adafruit_Circuit_Playground.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Me project is becomming challenging now :smiling_face_with_tear:

Besides giving up, reschooling and becoming a train driver what else should I do now.

I have always had the IDE on the side and I believe IDE and CLI share the same library folders and what not.

Regards :tumbler_glass:

Bas

Arduino CLI uses the libraries installed under the libraries subfolder of the path configured by the directories.user configuration key. The default value of directories.user is C:\Users\<username>\Documents\Arduino but of course you can customize the configuration just as you did with directories.data.

It is "an approach". However, bundling the libraries with the sketch may require making modifications to the #include directives in the libraries in addition to the sketch. In this case, the error was caused by a bug in the "Adafruit_Circuit_Playground" library. They should have written the #include directive for the local header file like this:

#include "Adafruit_Circuit_Playground.h"

The bug doesn't make any difference when the library is installed normally, which is why the developer got away with this sloppy code, but then it comes out to bite you when you attempt to bundle it with the sketch.

Although there are valid use cases for bundling libraries with a sketch, I don't see any reason to do it in this particular case. As I mentioned above, it is possible to point Arduino CLI to a custom location to get libraries via the directories.user configuration key. There is another option, which is to specify the path to the folder that contains the libraries via the --libraries flag of the arduino-cli compile command:

https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/

Arduino IDE 1.x has three different libraries folders:

  • <directories.user>/libraries (in the sketchbook folder)
  • The "platform bundled" libraries of the currently selected board
  • The "built-in" libraries in the libraries subfolder of the Arduino IDE installation folder.

Arduino CLI has two different libraries folders by default:

The "Servo" library is installed in the "built-in" libraries folder by default. You can also install it to <directories.user>/libraries using the Arduino IDE Library Manager or arduino-cli lib install.

Okay, I just ran

./Arduino-cli/arduino-cli.exe lib install servo

change the #include back to <Servo.h> and suddenly I got a succesfull compilation on my hands.

I think that a bare installation of arduino-CLI.exe followed by this script (which does all the needed things) is the easiest approach.

echo off

set ARDUINO_DIRECTORIES_DATA=%~dp0\Arduino-cli\configFiles
Arduino-CLI\arduino-cli config init
Arduino-CLI\arduino-cli core update-index
Arduino-CLI\arduino-cli core install arduino:avr
Arduino-CLI\arduino-cli lib install servo
pause

I deleted the entire configFolder's content, ran the above script than run this script

echo off

Arduino-cli\arduino-cli compile -b arduino:avr:uno .\arduinoProgram
Arduino-cli\arduino-cli upload ./arduinoProgram -b arduino:avr:uno -p COM4

pause

And this resulted in a succesful upload. So so far so good. I am currently trying to call the flash batch file from me processing application. But the correct function and syntax currently eludes me. I am not sure to use open(), exec() or launch().

Good question for processing forum :wink:

When that works, I can prep an USB stick and test the scripts and binaries on a 'virgin' PC. :crossed_fingers:

Kind regards and thanks alot for the help so far :innocent:

Bas

Great job getting it to a working state!

My only concern is this will cause the Servo library to be installed at the default location under the user's "Documents" folder, something like this:

C:\Users\<username>\Documents\Arduino\libraries\Servo

That is completely fine on the system of someone who is writing Arduino sketches, but a user who doesn't know or care anything about Arduino would only consider it pollution of their Documents folder. They won't be sure where this Arduino folder came from or whether they can safely delete it.

In some rare cases, this location can be problematic. The problems are usually associated with the fact that Windows puts the "Documents" folder on Microsoft OneDrive if the user has enabled that service (and Microsoft takes every opportunity to coerce Windows users into using OneDrive so it is likely the majority of Windows users have it enabled).

  • The "Documents" folder might not be accessible if the user is not connected to the OneDrive service or if something was misconfigured when they uninstalled the service after having enabled it.
  • The folder name is localized and the compiler is unfortunately not able to recognize libraries that are under paths that contain certain characters found in some paths that result from this localization.

One other thing you might consider is "pinning" the version of the dependencies of your project (arduino:avr platform, "Servo" library). Your current script will install the latest version of the dependencies. The benefit of that approach is that the compilation will always benefit from any bug fixes or enhancements made during the recent development in the dependencies. So your current approach is perfectly valid. The disadvantage is that a script that produces a perfect compilation today might stop working tomorrow if a release of a dependency was made with incompatible changes or a bug.

If you specify the version number that you have verified as working in the installation command, then the stability of the script is increased:

Arduino-CLI\arduino-cli lib install Servo@1.2.1

If a new version of the "Servo" library dependency is released, you can validate it at your convenience and then update the version number in the script only after confirming that the version bump is compatible with your sketch.

Now you mention... My wife's laptop started to complain about memory even though there were many many GB still free on the C disc.

It turned out that half that PC was coupled to one drive. I could not safely delete the one drive folder, that would delete.. everything and for good. So I simply disabled the one-drive "feature" (which essentially made the PC useless)

To me that one-drive is more like a computer virus, just like Yahoo and Bing :joy: #browserhijackers

If I would want a PC to use cloud storage, I would get one of those crappy google contraptions :joy:

Anyways back on-topic.

Can you not configure Arduino-CLI to use a different library folder location before you install the libraries?

I am running out of ideas by now. I think that letting the library install in the document folder as it is now, may be the least bad alternative? So if you have any other viable suggestions...

Perhaps using an alternative servo library with less dependancies may work? Which I can locally "include"

Arduino-CLI\arduino-cli lib install Servo@1.2.1

Well noted :+1:

Yes. I explained it already in post #11.

Libraries are installed to the libraries subfolder of the path configured via the directories.user configuration key. So you do via the directories.user configuration key.

I suggested two solutions already in post #11. Pick whichever one of the two you prefer:


If you want to install the library on demand, use the directories.user configuration key to set a custom location for arduino-cli lib install to install the library.


If you want to bundle the library with a complete package you distribute to the users, then add the library files to any convenient location in the package and then pass the path to the location of the pre-installed library in the distribution to a --library flag in your arduino-cli compile command.


The official "Servo" library doesn't have any dependencies.

Did you interpret this error you experienced as indicating that the "Adafruit_Circuit_Playground" library was a dependency of the "Servo" library?:

If so, that was an incorrect interpretation.

The only reason you encountered this error was because you put the "Adafruit_Circuit_Playground" library under the src subfolder of the sketch. The Arduino sketch build system compiles all the source files under the src subfolder (recursively) so if you throw a library in there, then any bug or incompatibility in the library is going to cause a compilation error even if your sketch program doesn't use the library.

1 Like

I am sorry, I got confused about the DATA and USER directory. In post #13 I posted this. I do set the directories.data like you suggested.

I thought I was doing this right, than you made your comment about the documents folders and one-drive related issues. That got me confused.

But I think I understand now. Besides adding this line

set ARDUINO_DIRECTORIES_DATA=%~dp0\Arduino-cli\configFiles

I also should have added this line.

set ARDUINO_DIRECTORIES_USER=%~dp0\Arduino-cli\configFiles

I added this line to the install script, ran it and now it seems to install the servo libary where I want it.

On a side note,

I also managed to fix the batch file for flashing the program to a board. I had some difficulties getting the path names correct. It worked by double clicking, but not when invoking the script from processing.

This script now also works when invoked by processing.

echo off

echo showping path
SET mypath=%~dp0

echo "BUILDING"
%mypath:~0,-1%\Arduino-cli\arduino-cli compile -b arduino:avr:uno %mypath:~0,-1%\arduinoProgram

echo "UPLOADING"
%mypath:~0,-1%\Arduino-cli\arduino-cli upload %mypath:~0,-1%\arduinoProgram -b arduino:avr:uno -p COM4

echo "UPLOADING COMPLETE!"

sleep 2

%mypath:~0,-1% -> this batch-nese syntax is just :smiling_face_with_tear:

The last part of the puzzle is adding the com port and board names as parameters. Than I can finally deploy my 'IDE' with the new 'built-in' compiler. This will surely help model railway people with their arduino projects.

When it is finished, I'll post it on this forum as well. I still have to translate my manual to English manual and add english videos as well.

Kind regards :tumbler_glass:

Bas

Nice!

Yeah, I've been working with batch files for 30 years and I still don't have anything like fluency in the language. Fortunately I switched to using the Git Bash shell on my Windows machine some years back (just before I would have probably gone down the rabbit hole of PowerShell scripts) and so haven't had to write batch files much since. But of course when it comes to supporting the average Windows user, they won't have a Posix shell set up and would be unable/unwilling/reluctant to set one up, so we must make do with what is available on their system already.

You might be interested in arduino-cli board list for this purpose. It lists all the ports found on the system, and identifies them as Arduino boards when possible.

If you add the --format json flag to the command, the output is in machine readable JSON. You might be able to leverage this in your Processing program to provide the user with a nice interface for selecting the port.

Very cool! Thanks.

I am familiar of the board list command. I am thinking... my flash.bat only calls arduino-cli build followed by an upload. If I can simply launch the commands directly I could omit the batch file entirly.

I'll first try to figure out how to fetch the respons from the boards list command (or any other command) in processing.

For those who might be interested. I managed to interact with arduino-cli directly via processing. I had some troubles because I didn't know I had to import some libraries.. which was obvious in retrospect...

I am currently assembling this function to fetch board data. I must now learn how to parse the JSON data. I believe it isn't that hard. Did it once in python. no big deal.

void flashProgram()
{
    String myPath = sketchPath() ;
    String boardName ;
    String COM_PORT ;
    String command ;
    String line ;

    String arduinoCliPath = myPath + "\\Arduino-cli\\arduino-cli.exe " ;
    String sketchPath     = myPath + "arduinoProgram" ;
    String listCommand    = "board list --format json" ;

    try
    {
        command = arduinoCliPath + listCommand ;
        Process p = launch(command);
        BufferedReader in = new BufferedReader(new InputStreamReader( p.getInputStream()));
 
        while ((line = in.readLine(  )) != null) println(line);
    } 
    catch (RuntimeException e) {println("RuntimeException, it fails") ; }
    catch (IOException e) {println("IOException, it fails") ; }

The output seems legit:

"matching_boards": [
      {
        "name": "Arduino Uno",
        "fqbn": "arduino:avr:uno"
      }
    ],
    "port": {
      "address": "COM4",
      "label": "COM4",
      "protocol": "serial",
      "protocol_label": "Serial Port (USB)",
      "properties": {
        "pid": "0x0043",
        "serialNumber": "85735313333351612150",
        "vid": "0x2341"

I have one relevant question. I am testing with an original arduino here. If I or some other guy uses one of those chinese things with ch340 drivers, can I still destinct an uno from a nano?

I don't know how board listall works under the hood. I am not sure if the bootloader tells these things to the computer or the usb-serial chip does. I suspect the latter?

Kind regards :coffee:

Bas