Helping the Arduino build environment grow up

Let me start this off by saying that I think the Arduino team has done a wonderful job lowering the bar to entry to microcontroller development for people with little to no software development experience. The IDE is impressively simple, streamlined, and user friendly and is a great way to get your foot in the door. As someone who's been doing software development for at least 10 years, now, however, I quickly became frustrated with my persistent muscle memory when editing text in the IDE and found myself longing for my beloved TextMate editor. Within a day of receiving my Arduino board, I started trying to figure out how to use the Makefiles (a skill that I haven't used since '97 or so) and break up the rust on the parts of my brain that used to know how to program in C. I started to dissect the Makefile today and realized that there's a ton of reuse that can be achieved by making the build environment a little more (de-facto) standards-oriented.

To start with, I think that everything in hardware/cores/arduino could be packaged up into a single "libarduino.a" archive. By doing that, the entire Arduino core can be linked in to each sketch without having to recompile HardwareSerial.cpp, pins_arduino.c, etc. It would make the per-sketch build process simpler and faster, with fewer dependencies that probably won't change at all for most people.

Similarly, I think each of the libraries in hardware/libraries could benefit from having their own Makefile that generates their own libWhatever.a archive. With a slight rejiggering of the Arduino conventions, we could either import libraries into a sketch by including, for example, "<EEPROM/EEPROM.h>", thus minimizing the pre-compilation processing that has to be done to the .pde files.

With these two small(ish) changes, I think it'd be possible to move between the Arduino IDE and command-line Makefiles without having to modify the sketch code as is necessary today. It should be possible to link in the arduino core and EEPROM library with an appropriate -L flag and -larduino and -leeprom in the CFLAGS macro in the sketch's Makefile. The way it stands today, it seems that you're either using the IDE or you're using Makefiles, but not both, since the .pde files need pre-compilation processing that exists only in the Java code. Furthermore, as mentioned in one of the tutorials, it seems that there's a bug in the Arduino IDE code that prevents you from writing functions that take or return typdef'd types defined in a header file, since the function prototypes end up coming before the #include directive.

Is there sufficient interest to continue to pursue this publicly, or should I just keep tinkering with these for my own edification? I think I'm going to have some code that others will have interest in, but I dread having to maintain a version for myself alongside a version for the IDE.

Thanks,
Brian

Your suggestions aren't particularly compatible with the notion that tha hardware is somewhat mutable. You select between the an ATmega8 at 16MHz, ATmega168 at 16MHz, or ATmega168V at 8MHz in the arduino environment, and the library code is compiled to suit. Sure, there are ways around this in a traditional make-based environment, but...

(Of course, I've got a bit of a grudge against makefiles. The ones I WORK with every day, and most of those in the open source community, have reached a point where they're no longer comprehensible to anyone but a makefile wizard. I wouldn't like to see Arduino go that way...)

(this doesn't mean that you shouldn't keep experimenting. It'd be interesting to see just how much time could be saved.)

(I haven't found it particularly painful (so far) to edit in my favorite editor and pop into arduino for compile/download. Many microcontroller IDEs end up forcing you into this sort of model as well, not that this is a good thing.)

(I wonder what it sounds like when talking to a guy who speaks in parenthesis?) ;D

You're absolutely right on the hardware issue. I haven't examined that in detail, yet, but I think there are only two Makefile macros that would affect that: the target chip type, and the processor speed. The Arduino IDE manages that with a single drop-down menu with a fairly small set of targets. I figured I would just either have a recursive Makefile or a set of invocations of make that would generate a libarduino_$(TARGET).a file. You'd still need to have a per-sketch Makefile, and it would be up to the individual developer to set their target architecture, but if Arduino shipped with the required libraries, it would make their lives a little easier.

The decision to compile the core from source each time is intended to make it easier for people to tinker with the code - without having to set up a separate build process. No one seems to mind the extra second or two (or whatever it is).

Arduino 0011 will include a Makefile that should compile .pde sketches almost unchanged from the ones compiled by the IDE. The only difference is that it won't generate function prototypes automatically. Besides that, you shouldn't need to change the code to switch between the IDE and the Makefile. I don't think distributing a pre-compiled library would improve this compatibility.

On the other hand, feel free to show us what you come up with, and to keep making suggestions. There's lot of room for improvement.