How can I capture the output of the Arduino CLI?

Please note this is a cross-post of a question originally asked on Stack Overflow

I am experimenting with automating Arduino builds using Python's subprocess module. I cannot seem to capture the output from the compile/link process.

For example, if I call the Arduino CLI from my terminal using a sketch with a deliberate compiler error, I get this output:

james@james-pc:~/code/arduino-cli-test$ arduino --verify ~/sketchbook/test/test.ino
Picked up JAVA_TOOL_OPTIONS: 
Loading configuration...
Initialising packages...
Preparing boards...
Verifying...
sketch/test.cpp:7:25: fatal error: test.h: No such file or directory
 #include "test.h"
                  ^
compilation terminated.
exit status 1

But If I use subprocess to run the same command:

sketch_path="~/sketchbook/test/test.ino"
try:
    subprocess.run(["arduino", "--verify", sketch_path], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
    print(e.output)
    print(e.stdout)
    print(e.stderr)

The except block runs, but I only get the output from Arduino itself, not the compiler output:

Out: 
Std: 
Err: Picked up JAVA_TOOL_OPTIONS: 
Loading configuration...
Initialising packages...
Preparing boards...
Verifying...

In this instance, both e.output and e.stdout were blank. I get different results there depending on the stdout and stderr parameters, but no combination of parameters I've tried has resulted in capturing the compiler output.

I get the same result if I eliminate check=True and examine the returned CompletedProcess object.

Why is this the case? Is there a way I can capture this output?

Footnote: Looking at this github issue and reading the documentation, maybe this isn't possible as the "arduino cli" is not very cli-like under the hood. Also, even redirect to file doesn't work, it just prints to the terminal:

james@james-pc:~/code/arduino-cli-test$ arduino --verify ~/sketchbook/test/test.ino > test.txt
Picked up JAVA_TOOL_OPTIONS: 
Loading configuration...
Initialising packages...
Preparing boards...
Verifying...
sketch/test.cpp:7:25: fatal error: test.h: No such file or directory
 #include "test.h"
                  ^
compilation terminated.
exit status 1
james@james-pc:~/code/arduino-cli-test$ cat test.txt 
james@james-pc:~/code/arduino-cli-test$

Any assistance on automating the build would be appreciated. I could of course use my own makefile or other build process, but I had really hoped to use the Arduino executable as-is.

jamesfowkes:
I am experimenting with automating Arduino builds using Python's subprocess module. I cannot seem to capture the output from the compile/link process.

I don't understand from that what you are actually trying to do with Python.
Consequently I don't understand why you would want to capture the output.

I have written this simple Python program to compile and upload using the command-line IDE.

...R

I want to do exactly what you're doing, but along with capturing the return code, also capturing the compiler/link/upload output, so that in the case of a failure of one of those processes I can print/store the exact error that occurred.

That seems to be the hard bit.

Using Linux, errors are written to standard error so the following results in standard output in a file named stdout.txt and standard error output in a file named stderr.txt

arduino --verify mysketch/mysketch.ino >stdout.txt 2>stderr.txt

To get both in the same file named stdout.txt, do the following.

arduino --verify mysketch/mysketch.ino  >stdout.txt 2>&1

I do not know if this works on Windows or Mac or python.

jamesfowkes:
I want to do exactly what you're doing, but along with capturing the return code, also capturing the compiler/link/upload output, so that in the case of a failure of one of those processes I can print/store the exact error that occurred.

That seems to be the hard bit.

When I compile or upload an Arduino program with my Python script it opens a terminal in which all the usual compile/upload messages appear (just as they would in the Arduino IDE). If there is a problem I can read the text on screen. I have never felt the need to save the text for posterity.

...R

This is part of a project to automate builds/uploads, so the output needs to be logged/captured somehow.

As I mentioned earlier, my Python program causes a terminal window to open - I wonder if your program could give a command to the terminal program to save its output to the file?

...R

Unbelievably, this was somehow due to using ~ for my home directory. Replacing that with a fully qualified path did the trick.

Thanks for the update.

For the benefit of others can you post your working version?

...R