Suggestion for making the Arduino environment more useful for commandline users

Development Suggestions for the Arduino Project
1

There are many messages about how to use Makefiles with arduino projects, so that the sketch can be compiled without starting the IDE. Responses vary between "I don't have a problem using the IDE" (which doesn't actually answer the question?) and "google for a Makefile and copy it into your source tree". Neither of those is a proper solution, IMO.

I'll first explain what I consider the issues to be fixed:

Firstly, there is the problem of duplicated code. If people have to copy the Makefile into their source, there will be thousands of copies all over the place. This is in fact the current status. And many of those copies have once been made, but are not maintained. This results in code which doesn't work with the new Arduino code (in particular the change to 1.0 has caused a lot of breakage IME).

The old solution of including a Makefile in the Arduino tree which should be copied into your project suffers from this problem, and is therefore not a good idea.

I'm not arguing to freeze the API. I'm arguing to avoid code duplication. (I'll get to a method for it in a moment.) Without code duplication, there is no problem: the code is updated with the API and everything continues to work.

Secondly, there is the issue of error messages. This is actually a problem which is unrelated to the Makefile issue, but it is easily solved at the same time. What problem am I talking about? That line numbers in error messages are incorrect, because they refer to a generated file instead of the actual source. There are two proper ways to solve this, which I'll explain below.

So, what do I propose as a solution? I propose to use a standardized process for generating the source code to be compiled. This process must be implemented in a commandline tool. I do not care how it is done, as long as it can be done without opening (and preferably without even installing) the IDE. Everything that the IDE currently does between the moment the user clicks on "verify" or "upload" until the spawning of the compiler process should be put in a different executable which can be separately called. The IDE will then call that executable to prepare the code, but other programs can also call it. In fact, it would be good if that process would also call the compiler (but people will probably like it if calling the compiler can be disabled with a commandline flag). It is important that the IDE itself calls it, because that means that it is impossible to get bugs from the IDE source and the code-preparing source getting out of sync (in fact, this is the same argument as before: no code duplication).

Once this tool is available, Makefiles can simply call it instead of the C++ compiler, and be as simple as for other programs. It would be nice if the tool also supported an "upload" option, so that Makefiles don't need to duplicate that functionality either. Again, the IDE must then use this tool for uploading as well, to avoid them getting out of sync.

I hope you like the above proposal. Assuming that you do for a moment, I'll continue and propose a change to how the source is transformed to real C++ code.

As I wrote, the problem of error messages can be solved in two ways:

  • Add information about source file and line numbers to the generated files. That way, the compiler will know how to report the errors. How to do this, is however compiler-specific. I suppose that doesn't really matter, because the code only needs to run on one compiler anyway. In gcc, it is done by adding lines of the form:
    #line <num> "<file>"
    (where and are substituted). The line immediately after that line is considered line number in file by the compiler, irrespective of its actual source file and line number. It will use the provided info in error messages. It will simply increment its line number until another #line directive is encountered.[/li]
  • Don't change the file, but create a wrapper which #includes the *.ino file. That way the compiler will obviously also know which line and file it is, because it uses the source directly.

Personally, I'd go with the latter, but either will work and neither is bad.

Finally, if I can be of service implementing any of this, don't hesitate to ask. (However, I can read but not write java, so I'm not sure how helpful I can be.)

Thanks,
Bas

2

I believe that the specific features you have requested:

  1. provide a command-line that generates the pre-processed source.
  2. fix the line number correspondence.
    have both been implemented in more recent versions of the IDE.

Without code duplication, there is no problem: the code is updated with the API and everything continues to work.

There is still a problem, because someone still has to update the new code functionality. Just because that functionality is part of the official distribution doesn't mean that it becomes effortless to maintain. The Arduino "issue" list is long, and maintaining odd features for people outside of the arduino target audience (like makefile compatibility) is a lower priority. For example, the line number fix introduced a bug in avr-objdump disassembly output...

3

westfw:
I believe that the specific features you have requested:

  1. provide a command-line that generates the pre-processed source.
  2. fix the line number correspondence.
    have both been implemented in more recent versions of the IDE.

Oh, cool. I didn't realize I was using an older version, sorry about that.

Without code duplication, there is no problem: the code is updated with the API and everything continues to work.

There is still a problem, because someone still has to update the new code functionality. Just because that functionality is part of the official distribution doesn't mean that it becomes effortless to maintain.

You misunderstand me. What I'm saying is that if there is only one place where a thing is implemented, then if someone wants to change the functionality, they automatically change it in all places. Because if they didn't change it in that one place, then they didn't change anything.

So sure, maintenance is still required. But without code copies, it is impossible for parts to get out of sync.

The Arduino "issue" list is long, and maintaining odd features for people outside of the arduino target audience (like makefile compatibility) is a lower priority.

I fully understand. That's why I suggested to make this part of the main process, so it isn't an "odd" feature. But it seems I was a bit too late with my suggestion, and you have already done exactly that. :slight_smile:

Thanks,
Bas

4

westfw:
For example, the line number fix introduced a bug in avr-objdump disassembly output...

That's probably my fault, since I contributed the #line code. It did fix long-standing bugs with highlighting errors when using multiple tabs. I did not know about the unintended consequences with objdump. In fact, I still don't really know the details. Any chance you could take a moment to explain?

On my to-do list (admittedly not near the top) is further improvement to how Arduino parses the compiler errors. There are still lots of cases where Arduino gets confused by the compiler output and fails to highlight the correct line. I plan to fix some of those cases. Those sorts of fixes are relatively easy to get accepted to Arduino. I suspect the Arduino Team does not care at all about objdump formatting... but if the fix were automatically part of improved error parsing it might get used?

First I need to understand better what when wrong with objdump?

5

Any chance you could take a moment to explain?

I did submit #line directive interferes with disassembly. · Issue #1337 · arduino/Arduino · GitHub
Basically, the preprocessed file (in the build directory) gets lines like "#line 1 Blink.ino", but "blink.ino" is off in the original sketch directory, "out of scope" for the simple forms of avr-objdump.

6

Thanks. That's the info I needed. Looks like I'm not following Arduino's issue tracker so closely these days.....

I've put this on my to-do list. It's a low priority, but unless someone else fixes it first (very unlikely), I'll do it eventually.

7

Even easier...

:slight_smile: