Option to bypass "Processing -> C++" pre-processor

I would like to see an option to turn off/byapps the "Processing -> C++" preprocessor, and instead directly write the main() function, include my own prototype declarations, header files, calls to any required intialization functions, etc.

You might think "well, just ditch the Arduino IDE and go directly to avr-gcc", but the cost in doing that is that you lose access to the great wealth of Arduino libraries written by third parties, many/most of which rely on the "wiring" Arduino core libs to some degree.

So I'd ideally like to stay in the Arduino library "ecosystem", just opting out of writing "sketches" in "Processing", and instead writing C++ programs directly all the way through. I already write a considerable amount of C++ for my Arduino programs when I decide it's time to package certain things up into an Arduino "library" anyway -- I'd just like the option of doing this all the way through.

I already use my own editor by selecting the "use external editor" feature -- I really appreciate being able to do that.

Why do I wish to be able to disable the "Processing" preprocessor? Mostly because, for me, I find the benefits are far outweighed by the drawbacks. I've been bitten too many times by bugs that have me wasting way too much time chasing nonsensical compiler errors. The inability of the "Processing -> C++" preprocessor to even manage standard C/C++ conditional preprocessor directives is well documented. This makes configuration management a pain, and while the Arduino development team are well aware of the issue, it has been made clear it's not on the list for fixes (sounds like the preprocessor regular expression code has gotten to the stage that no-one fully understands exactly what it's doing anymore) -- so it looks like it's not going to get fixed anytime soon.

But I've come across weirder and harder to understand (and therefore workaround) bugs -- most recently I had some code that would throw up spurious errors (some name collision of some sort going on under the hood, I suspect), but would compile without error when the offending code was placed in a seperate .h file and then inlined in the sketch in the same place via a #include directive.

That just about did it for me.

So I'm just at the stage where I really think I I'd like to leave "Processing" behind me (except perhaps for really quick and dirty apps where the typing time is actually a significant part of the development time.)

Would anyone else like to see turning off the "Processing -> C++" preprocessing as an option within the Arduino toolchain? I appreciate the simplifications that "Processing" offers over real C++ is a boon for programming novices, but not everyone who programs Arduinos is a programming novice, and one size doesn't fit all. I'd very much like the option to be able to trade off the costs and benefits of using "Processing" on a case by case basis -- seeing an extra checkbox under "Use external editor" that says "Use C++ instead of Processing" would be a very welcome feature indeed.

Alternatively, anyone who has been down this path have any advice on how to proceed sidestepping "Processing" without throwing the Arduino baby out with the bathwater (so to speak)? Thoughts and discussion appreciated.

You can write your own "main", eg. this compiles:

int main ()
 {
 return 0;
 }

You could make the "processing" sketch (the .ino file) simply have the includes for any libraries you want, and then use .cpp files for the rest of the project. That way the preprocessor's affect is minimal.

I recently switched to using a makefile. I use this one: GitHub - sudar/Arduino-Makefile: Makefile for Arduino sketches. It defines the workflows for compiling code, flashing it to Arduino and even communicating through Serial.

This means:

  • No GUI! yay
  • I get to use vim!
  • It automatically links in the arduino libraries and it’s easy to tell it to link in other ones like Servo or AFMotor
  • It only addes the #include <Arduino.h> to your sketch
  • It only recompiles needed files
  • You can build, upload, and monitor the serial port all with one command (make upload monitor)[/ii]

The arduino core still provides the main function, but you can still override it with your own.

It would be nice to be able to conditionally include HardwareSerial so you don’t have to use the extra bytes for the preinstated HardwareSerial object. Can you force the linker to ignore it if you, for example, #define HardwareSerial_h before you include Arduino.h? it might notice nothing is being used from HardwareSerial.cpp

the cost in doing that is that you lose access to the great wealth of Arduino libraries written by third parties, many/most of which rely on the “wiring” Arduino core libs to some degree.

So, keep using the core library. Everything needed to compile without the IDE is included with the IDE.

WizenedEE:
I recently switched to using a makefile. I use this one: GitHub - sudar/Arduino-Makefile: Makefile for Arduino sketches. It defines the workflows for compiling code, flashing it to Arduino and even communicating through Serial.

It would be nice to be able to conditionally include HardwareSerial so you don’t have to use the extra bytes for the preinstated HardwareSerial object. Can you force the linker to ignore it if you, for example, #define HardwareSerial_h before you include Arduino.h? it might notice nothing is being used from HardwareSerial.cpp

Yeah, I’ve been thinking about switching from the IDE to using normal Makefiles and emacs. Thanks for the link.

I haven’t looked at the Makefile yet, but the standard way of not bringing in things like HardwareSerial, is to put them in a library such as libhwserial.a with the avr-ar and avr-ranlib commands. Then use -L -lhwserial on the link line where is where you have stored the .a files.

More advanced use would build everything with the -fdata-sections -ffunction-sections, and on the link line add -Wl,–gc-sections. What this does is put every function and global variable into separate linker sections, and the linker would then eliminate sections that have no references. So, if you included the Hardware Serial object but no one ever called it, it would be eliminated from the final executable.

WizenedEE: I recently switched to using a makefile. I use this one: https://github.com/sudar/Arduino-Makefile

I just came back to check on the responses here. In the meantime, I've been hacking away getting Martin Oldfield's Arduino makefile v0.8 running under Win32 using the "make" that comes with MSys 1.0, and his ard-parse-boards perl script using Strawberry perl. It's working now, at least with the basic example sketches I've tested it with. So Yay! indeed. Emacs + makefiles means no more Processing. :-) Managing my own header files and prototypes is a very small price to pay to have the C/C++ preprocessor working properly again. All simpler than I thought it would be, actually.

It seems that everyone has converged on the makefile solution as the way to go. Kudos to Martin Oldfield and others for their work on this!

MichaelMeissner: More advanced use would build everything with the -fdata-sections -ffunction-sections, and on the link line add -Wl,--gc-sections. What this does is put every function and global variable into separate linker sections, and the linker would then eliminate sections that have no references. So, if you included the Hardware Serial object but no one ever called it, it would be eliminated from the final executable.

I had a look at Martin's makefile, and these are the options he uses by default as specified in CPPFLAGS and LDFLAGS. So that's nice.

The final thing I'd like to do at some stage is to get the Arduino "libraries" set-up as true link libraries. Setting up a scheme where each target platform (Uno, Duemil, the two Megas, etc.) would have it's own .a file for each library. Each .a file would get built by make on demand if it id didn't already exist the first time it was needed (or rebuilt if any of the dependencies had been changed, of course). So you could have

$(ARDUINO_DIR)/libraries/DS1307/uno/DS1307.a

and

$(ARDUINO_DIR)/libraries/DS1307/mega2560/DS1307.a

for example.

Seems sensible and obvious. (Unless anyone can think of a reason this won't work.)

Martin Oldfield's scheme in v0.8 builds the .o files in $(SKETCH_DIR)/build-cli/libs/DS1307/DS1307.o for example. Which is consistent with the Arduino IDE approach of compiling Arduino "library" files as if they are just additional source files belonging to the current project, so I suppose that's why he's gone that way.

Nothing I can't live with for the time being, though.

It's fast. It's simple. It's transparent. And eliminates an entire class of bugs (usually just called "Processing", LOL). Really, what's not to like?

I'm a much happier Arduino camper than I was about 24 hours ago! :-)

get the Arduino "libraries" set-up as true link libraries.

Is there a reason to do this in addition to the "-ffunction-sections" "--gc-sections" options?

Do set the Arduino IDE to "verbose" mode and watch what it does before you go off trying to create a makefile from scratch...

westfw:

get the Arduino "libraries" set-up as true link libraries.

Is there a reason to do this in addition to the "-ffunction-sections" "--gc-sections" options?

Mostly just to avoid the unnecessary recompilations, I guess. The "make" way of doing things is to rationally manage a build so things don't get recompiled unless they actually need to be, after all. So a little bit of a speed of build benefit, and probably a bit more satisfaction that things are being done "properly", if you will. There is some happiness to be had in having a sharp set of tools. But as you point out, perhaps no difference in terms of what the .hex ultimately looks like.

westfw: Do set the Arduino IDE to "verbose" mode and watch what it does before you go off trying to create a makefile from scratch...

Good advice -- I will certainly do this if I proceed. But in the near-term, I'll go with the system as it is, in order to get a bit of confidence it has indeed covered all the bases as well as it appears.

For completeness, I should post a link to Martin Oldfield’s web page:

there are also appear to be a few forks from this to check out if interested e.g.

as well as the one by Sudar mentioned above (neither of which I’ve played with yet.)

Latest cool thing I’ve discovered: If you want to use your own main.cpp rather than the default “core” version supplied in the IDE distro, easy peasy: Just create a source file in your sketch directory named main.cpp, and will be used to build main.o rather than the use the “core” source file. If main.cpp isn’t found, the default one will be used instead. Not sure if this is intended behaviour or just a fortunate side-effect of the order in which the make recipes are executed, but cool nonetheless.

The only thing that has to be in your main.cpp if you want to use the wiring libs is #include <WProgram.h> (or <Arduino.h>), and call init() before calling setup() (or however you decide to organise your main() function.)

Of course, also include any prototypes if you are calling anything beside init(), setup() and loop(), e,g.:

#include <WProgram.h>

void setup2();

int main(void)
{
	init();

	setup2();
    
	for (;;)
		loop();
        
	return 0;
}

(this was my test alternative main.cpp to see if it all works as expected.)

Anyway, this is all too great. Using my own editor and the makefiles, there isn’t any need to even fire up the IDE at all (unless I feel the need for a quick and dirty in “Processing” I suppose. But since I usually have emacs running anyway, I can’t see this happening very often, somehow…)

Woot!

unless I feel the need for a quick and dirty in "Processing" I suppose

Just a nit: it's NOT "Processing." Processing is a desktop programming environment based on Java in about the same way that the Arduino IDE is based on C++ (http://processing.org ) I don't think that there's a name for the IDE pre-processing that occurs on your .ino files and whatnot. "Arduino IDE preprocessing" is probably the most accurate phrase...

westfw:

unless I feel the need for a quick and dirty in "Processing" I suppose

Just a nit: it's NOT "Processing." Processing is a desktop programming environment based on Java in about the same way that the Arduino IDE is based on C++ (http://processing.org ) I don't think that there's a name for the IDE pre-processing that occurs on your .ino files and whatnot. "Arduino IDE preprocessing" is probably the most accurate phrase...

Thanks for the clarification. I thought Arduino had simply adopted "Processing" as their beginners' language for the IDE. Is this just me or this a common misperception? I don't honestly even remember hearing about either "Processing" or "Wiring" before becoming acquainted with Arduinos.

Unfortunately, "Arduino IDE preprocessing" is a bit of a mouthful. So I think I shall adopt the Victorian approach and simply declare "and from this day forth we shall never speak of this again".

It's a common misconception that "Arduino" is some kind of specialized language, rather than pretty straight C++ with a bit of pre-processing applied. About all that really happens is automatic prototype generation and concatenation of the .ino files (rather than compiling each separately.) (Unfortunately, the IDE is a little clumsy about exactly where those prototypes should be added, leading to annoying incompatibilities with some common C preprocessor-directive styles...)

westfw: It's a common misconception that "Arduino" is some kind of specialized language, rather than pretty straight C++ with a bit of pre-processing applied. About all that really happens is automatic prototype generation and concatenation of the .ino files (rather than compiling each separately.) (Unfortunately, the IDE is a little clumsy about exactly where those prototypes should be added, leading to annoying incompatibilities with some common C preprocessor-directive styles...)

It also attempts to manage header files for you, which is where I suspect most of the bugs come from.

I was also under the impression it was pretty heavily subsetted version of C++ down to not much more than straight C (which of course id a proper subset of C++). And of course, a "sketch" being limited to one source file. Indeed, about the only hint that C++ is actually the underlying compiler rather than C is that class method reference constructions such as Serial.begin() are allowed. I don't think you can actually define classes within the that-which-must-not-be-named though (although I've not tried, I just went to a creating "library" *.cpp file for that whenever needed.)

FWIW, I've never really been convinced that automatically generating prototypes or managing header file inclusions are that much of a big deal that would constitute much of a benefit to novice programmers. Maybe the strict subsetting is helpful, I don't know. Is it useful or counterproductive to force beginners to put everything into one pde or ino file? Again, I don't know.

In any case, I'm just glad to have it behind me, whatever the hell that was.

I mention in the sticky here that the underlying language is C++. I think it's an important point to make (which is why it is point #1).

With the exception of exceptions (lol) most other things can be done, and done easily. For example, you can use the STL if you download Andy Brown's port of it (The Standard Template Library (STL) for AVR with C++ streams ).

If you can use that (which relies heavily on templates) you can do pretty-much anything.

So you can define your own classes in a pde/ino file? As I say, I never tried, but I was under the impression you had to create a seperate .cpp file to do that.

Whatever it is, it really aint C++ if you can’t define classes.

Well you can, as this illustrates:

typedef class
  {
  public:
    int bar;
  } tFoo;

tFoo a;

void handler ()
  {
  a.bar = 42;  
  }
  
void setup ()
{
  handler ();
}

void loop () {}

However there is a problem if you try to pass your own defined class to a function, because of the automatic function prototype generation, which puts the prototypes before where the type is defined.

I just raised a bug report about that a few minutes ago:

http://code.google.com/p/arduino/issues/detail?id=973

On the bright side, if I may say without sounding pompous, most people who define a class in C++ would put it into its own .h files. That is really where class definitions belong. Then they can be shared between multiple .cpp files. You can do that under the Arduino IDE (making a new tab), it compiles correctly, and is easy to work with.

Whatever it is, it really aint C++ if you can't define classes.

Yep, you can define classes.

OK, I suppose I should have said define and use. LOL.

Sort of like a write-only memory device. ;-)

In any case, I'm glad to be in C++. No questions of "pretty much" or "bug reports".

I would disagree. Or say that if indeed most people are doing it this way, they are doing it wrong. You put the declarations (basically, prototypes for the class methods) into a .h file, but the the source for the methods is in a .cpp file. You definitely don’t want this included in every file that simply want to use a class!

pico: OK, I suppose I should have said define and use. LOL.

My example used the class. It compiles.

You put the declarations (basically, prototypes for the class methods) into a .h file, but the the source for the methods is in a .cpp file. You definitely don't want this included in every file that simply want to use a class!

I don't get your point here. You put the definition in a .h file. You can put the implementation into the .ino file, or make another .cpp tab for it. Probably neater to put it into a separate file. So say you want implement class foo, you make foo.h and foo.cpp. What is the major problem with that?

In any case, I'm glad to be in C++. No questions of "pretty much" or "bug reports".

Right. So C++ has no bug reports?

pico: But I've come across weirder and harder to understand (and therefore workaround) bugs -- most recently I had some code that would throw up spurious errors (some name collision of some sort going on under the hood, I suspect), but would compile without error when the offending code was placed in a seperate .h file and then inlined in the sketch in the same place via a #include directive.

That just about did it for me.

Perhaps you should calm down and post the code in question, and try to understand the techniques required to work around any issues you have. The IDE has some idiosyncrasies, designed to make things easier for beginners. Now you can throw out the whole lot, and move to a pure avr-gcc environment, or keep the friendliness of the IDE and develop strategies for avoiding the issues that friendliness causes.