Option to bypass "Processing -> C++" pre-processor

I mention in the sticky here that the underlying language is C++. I think it's an important point to make (which is why it is point #1).

With the exception of exceptions (lol) most other things can be done, and done easily. For example, you can use the STL if you download Andy Brown's port of it (The Standard Template Library (STL) for AVR with C++ streams ).

If you can use that (which relies heavily on templates) you can do pretty-much anything.

So you can define your own classes in a pde/ino file? As I say, I never tried, but I was under the impression you had to create a seperate .cpp file to do that.

Whatever it is, it really aint C++ if you can't define classes.

Well you can, as this illustrates:

typedef class
  {
  public:
    int bar;
  } tFoo;

tFoo a;
 
void handler ()
  {
  a.bar = 42;  
  }
  
void setup ()
{
  handler ();
}

void loop () {}

However there is a problem if you try to pass your own defined class to a function, because of the automatic function prototype generation, which puts the prototypes before where the type is defined.

I just raised a bug report about that a few minutes ago:

http://code.google.com/p/arduino/issues/detail?id=973

On the bright side, if I may say without sounding pompous, most people who define a class in C++ would put it into its own .h files. That is really where class definitions belong. Then they can be shared between multiple .cpp files. You can do that under the Arduino IDE (making a new tab), it compiles correctly, and is easy to work with.

Whatever it is, it really aint C++ if you can't define classes.

Yep, you can define classes.

OK, I suppose I should have said define and use. LOL.

Sort of like a write-only memory device. :wink:

In any case, I'm glad to be in C++. No questions of "pretty much" or "bug reports".

I would disagree. Or say that if indeed most people are doing it this way, they are doing it wrong. You put the declarations (basically, prototypes for the class methods) into a .h file, but the the source for the methods is in a .cpp file. You definitely don't want this included in every file that simply want to use a class!

pico:
OK, I suppose I should have said define and use. LOL.

My example used the class. It compiles.

You put the declarations (basically, prototypes for the class methods) into a .h file, but the the source for the methods is in a .cpp file. You definitely don't want this included in every file that simply want to use a class!

I don't get your point here. You put the definition in a .h file. You can put the implementation into the .ino file, or make another .cpp tab for it. Probably neater to put it into a separate file. So say you want implement class foo, you make foo.h and foo.cpp. What is the major problem with that?

In any case, I'm glad to be in C++. No questions of "pretty much" or "bug reports".

Right. So C++ has no bug reports?

pico:
But I've come across weirder and harder to understand (and therefore workaround) bugs -- most recently I had some code that would throw up spurious errors (some name collision of some sort going on under the hood, I suspect), but would compile without error when the offending code was placed in a seperate .h file and then inlined in the sketch in the same place via a #include directive.

That just about did it for me.

Perhaps you should calm down and post the code in question, and try to understand the techniques required to work around any issues you have. The IDE has some idiosyncrasies, designed to make things easier for beginners. Now you can throw out the whole lot, and move to a pure avr-gcc environment, or keep the friendliness of the IDE and develop strategies for avoiding the issues that friendliness causes.

Let me put it to you like this. If you move away from the IDE, perhaps use a custom Make file, or use the Atmel Studio itself, you will no doubt eliminate some of the things that are bugging you. But, and this is important, you will replace them with other things that will annoy you. There is no perfect development environment.

No, of course not. But as there are degrees of evil, there are degrees of imperfection.

I probably should say, just to establish a bit of context, that I've dealt with many, many different development environments over the years, ranging over all sorts of hardware. The Arduino IDE was in no way the worst I've dealt wth. But very far from the best as well. (In fact, to be honest, it really does rank down there in the suckiness stakes. But I have dealt with worse.)

On a more positive note, there are many things apart from the IDE that I like about the Arduinos very much. I love the bootloader and the plug-and-play USB programming model. (I grew up with the old school development boards that you spent about a week getting set-up to get to "hello world" with. Intel, then Motorola, then MIPs.) I like the Atmel mega chips. I love the wealth of 3rd party libraries that means that just about any device you can think of interfacing with an Arduino probably has at least some basic code already available for it.

Which is why, overall, I am an Arduino fan.

But I am not a programming novice. I've been programming in C before there was even ANSI C, much less C++. Arduinos are for beginners. Fine. But I see no reason they have to be only for beginners.

So no, I don't expect "perfect" in a development environment. But emacs and gcc and make is pretty damned good, IMHO -- at least it's always something I've gotten along with. That's not to say "perfect". But, as they say, even the Sun has its spots. :slight_smile:

I'm glad there are options for those with differing tastes. Perhaps we can agree to agree on this.

pico:
But I am not a programming novice. I've been programming in C before there was even ANSI C, much less C++. Arduinos are for beginners. Fine. But I see no reason they have to be only for beginners.

Speaking as somebody who has used C since 1977 or 1978 and was on the original ANSI X3J11 C standards committee, I tend to agree with you in terms of the IDE trying to simplify and protect users. But as you point out not everybody who programs Arduinos needs the training wheels. In terms of development environments, I tend to be old school, and still haven't warmed to any graphical IDE, and only use it as a compilation unit and bootloader.

Found the definitive workaround, as discussed earlier in the thread.

You find the IDE "friendly". I find it a buggy PITA. Let's agree to disagree. And be happy for each other that you can use the IDE, and I can choose not to.

MichaelMeissner:
In terms of development environments, I tend to be old school, and still haven't warmed to any graphical IDE, and only use it as a compilation unit and bootloader.

I suspect our tastes are similar, and I'm now very happy to report that I now don't even require the IDE for compilation or bootloading. Recommended.

BTW, good job on ANSI C.

pico:
Mostly just to avoid the unnecessary recompilations, I guess. The "make" way of doing things is to rationally manage a build so things don't get recompiled unless they actually need to be, after all. So a little bit of a speed of build benefit, and probably a bit more satisfaction that things are being done "properly", if you will.

Did you realize Arduino 1.0.1 does this? Seriously, it does. I know....

I developed the code, originally in November 2010, as part of a patch on issue 393. It was resubmitted 1 year later as issue 638.

http://code.google.com/p/arduino/issues/detail?id=638

Last December it was committed into Arduino's github repository, and eventually released as part of Arduino 1.0.1.

I'm sure that does nothing to sway you to prefer the IDE over vim & a makefile... but when talking about ways command-line based make is superior to the Arduino IDE, avoiding recompiling files based on timestamps and dependency analysis has been part of Teensyduino for 1.5 years and now, as of 1.0.1, is officially a feature of the Arduino IDE.

You can bypass the IDE's preprocessing by placing all your code in .cpp and .h files, instead of .ino or .pde. You still need to have at least 1 .ino or .pde file which has the same name as the directory it's in. But that file can be empty.

The Arduino IDE supports .cpp, .c, and .h files within your sketch. They are NOT preprocessed in any way. If you want to use the normal Arduino functions, you must add the #include <Arduino.h> line in each .cpp file.

Within those .cpp files, you can define classes and use templates. You can also use them in .ino files, but there's a pretty good chance the IDE's not-too-smart preprocessing step will attempt to extract global-scope function prototypes. But that preprocessing stuff is NEVER used on .cpp files.

Likewise, the .cpp files are NOT parsed for includes to infer which libraries to link into your code. So in that empty .ino file, you can place #include <Library.h> lines, merely as directives to the IDE's build system to tell it which libraries to link. Then you can include only the library headers needed in specific .cpp files which call those library classes.... so your other .cpp files remain free of namespace clutter.

Very few people seem to know this feature exists, but indeed the Arduino IDE does support real C++ compilation without troublesome preprocessing. You merely need to add the .cpp files to the directory, and then do your coding in those .cpp files instead of .ino files.

Of course, there are a couple C++ things you don't get, not because of the IDE, but due to limitations in the underlying AVR toolchain. The avr-gcc compiler does not support exceptions, and avr-libc does not provide the C++ stdlib classes & functions.

But you can bypass the preprocessing step. It's as easy as just creating .cpp files in the sketch directory!

Excellent info, thanks Paul for the tips and insights. My cup runneth over with options... :slight_smile:

Choice is good. And Arduino development with the option to bypass the "helpful" preprocessor is better.

Hopefully a few more people will now be alerted to these basic choices available. I'm sure I can't be the only one glad to get the "sketch" preprocessing out of the picture! :slight_smile:

I mentioned that in reply #1, but thanks Paul for putting it more forcefully.

Also thanks for the tip about the way that unnecessary file are not recompiled every time. The IDE keeps getting better!

I'll agree that the update to the IDE improves the Arduino build process,
But the IDE even with this improvement still doesn't completely avoid doing unnecessary things or do what can be done with Makefiles.
Yes the update can prevent rebuilding of certain files for a given sketch under certain circumstances,
but since the IDE doesn't build core or Arduino library archives and store them independently of IDE sketch build area,
the IDE must rebuild library archives when say changing board types or restarting the IDE.
Because of the build methodology, the IDE also doesn't (can't) remember the compiled objects across instances of the IDE
even for the same sketch.
i.e. bring up IDE, build a sketch, exit IDE, bring up IDE for the same sketch - EVERYTHING rebuilds.
Even in the cases where the IDE is smart enough to not to re-compile core objects, it still
re-archives them into a local core library.

Overall, it is a build methodology problem. The IDE falls down in certain advanced capabilities because of
the methodology used and it is very difficult if not impossible to push the IDE to be "smarter" do things
differently without having to actually modify the IDE JAVA code.

With makefiles it is possible to do things in a more traditional way like
create 1 core library and 1 set of true libraries for each board type
and with proper dependencies never have to rebuild them again no matter what sketch uses them
or across sketch builds unless there is a change to the actual library code.

Consider an IDE like Atmel's AVR studio. It's build methodology was to use build on top of Makefiles so
that it can do things like allow you to override everything and use your own makefiles.

What annoys me the most about the IDE is that while it works quite well for certain work flows
and for novices, some of its implementation decisions - most of which would not have affected novices work flow
or impact their ease of use, get in the way of or even prevent doing certain more advanced things.

An Arduino library being able to call another Arduino library is very problematic with the current IDE build methodology.
Yeah, there is a hack to allow it to work today, but it involves adding header files to the actual user sketch - which is a
kludge since the sketch really doesn't care about such sub library dependencies.
Another, being able to source level debug using other Atmel tools.
Those types of problems can go away if a slightly different build methodology were used.

So for me, the building/re-building of modules unnecessarily isn't my biggest concern, what concerns me more is that
certain things like conditional compilation, (sketch being able to control compilation options of libraries), libraries being
able to call other libraries, library code being able to detect and conditionally change based on board type
(not processor type, or having to poke around for "magic" defines), being able to source level debug are the killers.
Heck even the ARDUINO define for the core code revision is not available in a header file as the build methodology
assumed things were only going to be built with the IDE.
It would be nice if the IDE developers would at least consider the ability to build makefiles around their core libraries
for the more advanced uses that want to use Makefiles so that they can do things like source level debugging
as some of the way they are doing things make it more difficult for building Arduino sketches with Makefiles.

--- bill

Yes the update can prevent rebuilding of certain files for a given sketch under certain circumstances,
but since the IDE doesn't build core or Arduino library archives and store them independently of IDE sketch build area,
the IDE must rebuild library archives when say changing board types or restarting the IDE.

Make, ant, or other build tools can't reuse code between board changes either. The Arduino core and most libraries contain MANY #ifdef checks on the processor and clock frequency. When you change to a different board, all that code simply must be recompiled.

Compiled code is indeed not saved between IDE sessions. Is that really a big deal? Really?

Long ago, Arduino compiled code to an "applet" subdirectory within the sketch. The change to using a temporary directory was primarily motivated by Arduino's usage in schools with tightly locked down systems. One of the few places you can be absolutely certain is writable under even the most strict security is a temporary directory.

An Arduino library being able to call another Arduino library is very problematic with the current IDE build methodology. Yeah, there is a hack to allow it to work today, but it involves adding header files to the actual user sketch

This is issue #236. I will fix it. I would have fixed it about 6 weeks ago, had David not temporarily changed his mind on this issue.

... what concerns me more is that certain things like conditional compilation, (sketch being able to control compilation options of libraries), libraries being able to call other libraries, library code being able to detect and conditionally change based on board type (not processor type, or having to poke around for "magic" defines), being able to source level debug are the killers.

Are you sure that's it? I mean, really, if these things were all fixed, I have a strong feeling you and many, many others who don't like the IDE still wouldn't like it.

Except for source level debugging, these are all pretty simple things. Already it is possible for sketches to control library options using C++ templates. The USB Host Shield library is one that leverages this to great effect. Others do so much less elegantly with headers to inline some code (my own Encoder library would be a good example of the dirty way). A standard way to detect the actual board has been an open issue for a long time, and one I'd personally like to patch. So far, the Arduino developers have shown little interest, but I'm patient and persistent. As more boards come out, it will become more compelling. Eventually I will fix that. Only days ago David (finally) relented and will accept a fix for issue 236, and I will write it in several weeks when I'm done with my current project.

I'm sure there are LOTS of other little things you and many others will name as Arduino IDE deficiencies once all this stuff is fixed. The simple fact is Arduino is designed to be simple and NOT have lots of advanced options. There are always going to be lots of things present in AVR/Atmel Studio, Eclipse and Visual Studio that won't be in Arduino.

But I do agree, source level debugging is the big issue. Of course, it's an absolutely non-starter with AVR. Atmel absolutely will not provide the debug specs for their AVR chips. Nobody, other than Atmel, can reasonably work on this (short of impractical reverse engineering) as long as AVR is the platform.

Depends on how it is implemented. If the build system creates a library archive for each board type,
the library code for that board type is only ever built once or whenever the library code changes.

There could also be an implementation where Arduino libraries are "installed".
Where at install time, the library code is then built for all board types and archived into the appropriate library archives.

The Arduino IDE could be updated with all the "smarts" to do this kind of stuff as well
so it is more of a build methodology issue than a specific tool issue.

Compiled code is indeed not saved between IDE sessions. Is that really a big deal? Really?

I agree that it is not that big of a deal. I even stated as much earlier. I was simply pointing out (in a long winded fashion)
that the IDE even with the additional smarts is still not quite fully doing what Pico was wanting/asking for
as even with its additional smarts it still sometimes re-compiling modules that have not changed.
(For the record, I didn't think the dumb way of building everything every time was that bad)

But in the bigger picture I wish if the IDE were not going to really do things "right" with dependencies
then I kind of wish it would even be simpler/dumber and simply glomb everything together into one HUGE compile unit rather
than to try to pretend to deal with things like library archives.
Compile times wouldn't be any longer than what was being done before the "smart build" updates, and
if everything were glombed together there are other compiler optimizations that could be done.
For example, it would be possible to crush down things like digitalWrite() into single instructions
as the compiler could now have visibility into all the mapping tables. This would allow getting the optimized
AVR bit set/clr instructions when using constants without having to resort to using separate matching macros to map things
when possible.

Long ago, Arduino compiled code to an "applet" subdirectory within the sketch. The change to using a temporary directory was primarily motivated by Arduino's usage in schools with tightly locked down systems. One of the few places you can be absolutely certain is writable under even the most strict security is a temporary directory.

I'm familiar with that and understand why they did what they did.
I don't necessarily disagree with it given the state of the rest of the IDE build process, but
I believe that there are some other alternatives like creating build areas under users sketchbook area
that offer the potential for more advanced/smarter capabilities.

An Arduino library being able to call another Arduino library is very problematic with the current IDE build methodology. Yeah, there is a hack to allow it to work today, but it involves adding header files to the actual user sketch

This is issue #236. I will fix it. I would have fixed it about 6 weeks ago, had David not temporarily changed his mind on this issue.

... what concerns me more is that certain things like conditional compilation, (sketch being able to control compilation options of libraries), libraries being able to call other libraries, library code being able to detect and conditionally change based on board type (not processor type, or having to poke around for "magic" defines), being able to source level debug are the killers.

Are you sure that's it? I mean, really, if these things were all fixed, I have a strong feeling you and many, many others who don't like the IDE still wouldn't like it.


I'm sure there are LOTS of other little things you and many others will name as Arduino IDE deficiencies once all this stuff is fixed. The simple fact is Arduino is designed to be simple and NOT have lots of advanced options. There are always going to be lots of things present in AVR/Atmel Studio, Eclipse and Visual Studio that won't be in Arduino

But I don't believe that
"Simple" has to preclude eliminating advanced capabilities.

I'm not a whiner. What I do tend complain about is when ease of use simplifications come at the expense of restricting
or even precluding more advanced functional capabilities,
especially when it is do to implementation/design decisions that really don't impact the simpler
uses or cost additional resources to implement.
For example, in some situations you can go down path A or B and neither affects high level usability
or ease of use but perhaps one make its easier or harder to do more advanced things.
This is some of the things I've seen in Arduino not just in the IDE but also in some of the core libraries.
I draw a clear distinction between the IDE and the Arduino software environment.
From my outsiders point of view, the Arduino teams vision doesn't seem to include much thought about this distinction.
It seems to be very much everything "Arduino" and the IDE and the s/w environment (core code and libraries)
are all seen as one.
So there doesn't seem to be much thought given to making the s/w environment more friendly to more advanced users
or users that wish to do Arduino development but perhaps not with the IDE.

In other words take a look at issue #236. It is a great example that shows how things are inter related.
If there were a board type Arduino library archive created for all the "installed" libraries
there would be no issue trying to figure out which libraries to link in. You could simply
add the top level libraries directories to the include path which would allow each library to locate other libraries
header files by using #include <libname/libheader.h>,
then link against the Arduino library archive for the currently selected board type.
So things come back full circle.
If the build methodology created board type Arduino library archives, not only could unnecessary
compiles be avoided, but it also makes things like linking in library code and handling sub library
dependencies easier.

Not that it has to be done that way, but just an example how things are inter related and how
things cascade by certain decisions in the build methodology.

For me, is isn't so much what the IDE does or doesn't do. It is more the build methodology and
what it prevents or makes difficult, whether that is using the IDE or not using the IDE.
So for example, I think all the talk about building/rebuilding objects is pure noise.
That kind of stuff is not a functional issue. It does not prevent or preclude anything from being done.
(Makefiles can be written to be just as dumb)
However, look at a few of some of items the advanced users are wanting:

  • libraries calling other libraries
  • ability to easily identify board type at compile time
  • ability to change/modify compile/linker options
  • ability to set compiler defines on a per board type basis.
  • ability to do source level debugging
  • enable floating point libraries, like printf()
  • better/new define names for pins to avoid having to detect and adjust code based on board types.

Those are functional issues. And currently the only way to make many of these work
is to either to modify the IDE or jump to something like your own Makefiles.

MPIDE handles several of these things much better in this area than the Arduino IDE as you can, if you choose, have more control
over the compile & link parameters all the way down to the board type level.

So overall I do wish there was a bit more thought going on about the Arduino s/w development
build process to ensure that things are easier for building more complex Arduino projects.
And just because the project is more complex doesn't mean that this advanced capability is
directly exposed to the end user or that it is more complex for the novice user
to build it or use it.

But I do agree, source level debugging is the big issue. Of course, it's an absolutely non-starter with AVR. Atmel absolutely will not provide the debug specs for their AVR chips. Nobody, other than Atmel, can reasonably work on this (short of impractical reverse engineering) as long as AVR is the platform.

Not necessarily. You can do it purely in software without having to deal with any of their
debugging hardware or interfaces. I know how to do source level debugging using only a serial port with gdb on the AVR.
It does require stealing a hardware pin to steal an interrupt,
and a very small (hundred or so byte support code monitor) but it can be done.

--- bill

Not necessarily. You can do it purely in software without having to deal with any of their
debugging hardware or interfaces. I know how to do source level debugging using only a serial port with gdb on the AVR.
It does require stealing a hardware pin to steal an interrupt,
and a very small (hundred or so byte support code monitor) but it can be done.

Useful. Got any more detail on that?