Arduino Forum

Development => Suggestions for the Arduino Project => Topic started by: pico on Jul 01, 2012, 06:57 am

Title: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 01, 2012, 06:57 am
I would like to see an option to turn off/byapps the "Processing -> C++" preprocessor, and instead directly write the main() function,
include my own prototype declarations, header files, calls to any required intialization functions, etc.

You might think "well, just ditch the Arduino IDE and go directly to avr-gcc", but the cost in doing that is that you lose access to the great wealth of Arduino libraries written by third parties, many/most of which rely on the "wiring" Arduino core libs to some degree.

So I'd ideally like to stay in the Arduino library "ecosystem", just opting out of writing "sketches" in "Processing", and instead writing C++ programs directly all the way through. I already write a considerable amount of C++ for my Arduino programs when I decide it's time to package certain things up into an Arduino "library" anyway -- I'd just like the option of doing this all the way through.

I already use my own editor by selecting the "use external editor" feature -- I really appreciate being able to do that.

Why do I wish to be able to disable the "Processing" preprocessor? Mostly because, for me, I find the benefits are far outweighed by the drawbacks. I've been bitten too many times by bugs that have me wasting way too much time chasing nonsensical compiler errors. The inability of the "Processing -> C++" preprocessor to even manage standard C/C++ conditional preprocessor directives is well documented. This makes configuration management a pain, and while the Arduino development team are well aware of the issue, it has been made clear it's not on the list for fixes (sounds like the preprocessor regular expression code has gotten to the stage that no-one fully understands exactly what it's doing anymore) -- so it looks like it's not going to get fixed anytime soon.

But I've come across weirder and harder to understand (and therefore workaround) bugs -- most recently I had some code that would throw up spurious errors (some name collision of some sort going on under the hood, I suspect), but _would_ compile without error when the offending code was placed in a seperate .h file and then inlined in the sketch in the same place via a #include directive.

That just about did it for me.

So I'm just at the stage where I really think I I'd like to leave "Processing" behind me (except perhaps for really quick and dirty apps where the typing time is actually a significant part of the development time.)  

Would anyone else like to see turning off the "Processing -> C++" preprocessing as an option within the Arduino toolchain? I appreciate the simplifications that "Processing" offers over real C++ is a boon for programming novices, but not everyone who programs Arduinos is a programming novice, and one size doesn't fit all. I'd very much like the option to be able to trade off the costs and benefits of using "Processing" on a case by case basis -- seeing an extra checkbox under "Use external editor" that says "Use C++ instead of Processing" would be a very welcome feature indeed.

Alternatively, anyone who has been down this path have any advice on how to proceed sidestepping "Processing" without throwing the Arduino baby out with the bathwater (so to speak)? Thoughts and discussion appreciated.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: nickgammon on Jul 01, 2012, 07:24 am
You can write your own "main", eg. this compiles:

Code: [Select]
int main ()
{
return 0;
}


You could make the "processing" sketch (the .ino file) simply have the includes for any libraries you want, and then use .cpp files for the rest of the project. That way the preprocessor's affect is minimal.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: WizenedEE on Jul 01, 2012, 04:08 pm
I recently switched to using a makefile. I use this one: https://github.com/sudar/Arduino-Makefile

This means:



The arduino core still provides the main function, but you can still override it with your own.

It would be nice to be able to conditionally include HardwareSerial so you don't have to use the extra bytes for the preinstated HardwareSerial object. Can you force the linker to ignore it if you, for example, #define HardwareSerial_h before you include Arduino.h? it might notice nothing is being used from HardwareSerial.cpp
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: cmiyc on Jul 01, 2012, 05:16 pm
Quote
the cost in doing that is that you lose access to the great wealth of Arduino libraries written by third parties, many/most of which rely on the "wiring" Arduino core libs to some degree.

So, keep using the core library.  Everything needed to compile without the IDE is included with the IDE.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: MichaelMeissner on Jul 01, 2012, 07:20 pm

I recently switched to using a makefile. I use this one: https://github.com/sudar/Arduino-Makefile

It would be nice to be able to conditionally include HardwareSerial so you don't have to use the extra bytes for the preinstated HardwareSerial object. Can you force the linker to ignore it if you, for example, #define HardwareSerial_h before you include Arduino.h? it might notice nothing is being used from HardwareSerial.cpp

Yeah, I've been thinking about switching from the IDE to using normal Makefiles and emacs.  Thanks for the link.

I haven't looked at the Makefile yet, but the standard way of not bringing in things like HardwareSerial, is to put them in a library such as libhwserial.a with the avr-ar and avr-ranlib commands.  Then use -L<dir> -lhwserial on the link line where <dir> is where you have stored the .a files.

More advanced use would build everything with the -fdata-sections -ffunction-sections, and on the link line add -Wl,--gc-sections.  What this does is put every function and global variable into separate linker sections, and the linker would then eliminate sections that have no references.  So, if you included the Hardware Serial object but no one ever called it, it would be eliminated from the final executable.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 01, 2012, 08:52 pm

I recently switched to using a makefile. I use this one: https://github.com/sudar/Arduino-Makefile


I just came back to check on the responses here. In the meantime, I've been hacking away getting Martin Oldfield's Arduino makefile v0.8 running under Win32 using the "make" that comes with MSys 1.0, and his ard-parse-boards perl script using Strawberry perl. It's working now, at least with the basic example sketches I've tested it with.  So Yay! indeed. Emacs + makefiles means no more Processing. :-) Managing my own header files and prototypes is a _very_ small price to pay to have the C/C++ preprocessor working properly again. All simpler than I thought it would be, actually.

It seems that everyone has converged on the makefile solution as the way to go. Kudos to Martin Oldfield and others for their work on this!

Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 02, 2012, 06:21 am

More advanced use would build everything with the -fdata-sections -ffunction-sections, and on the link line add -Wl,--gc-sections.  What this does is put every function and global variable into separate linker sections, and the linker would then eliminate sections that have no references.  So, if you included the Hardware Serial object but no one ever called it, it would be eliminated from the final executable.


I had a look at Martin's makefile, and these are the options he uses by default as specified in CPPFLAGS and LDFLAGS. So that's nice.

The final thing I'd like to do at some stage is to get the Arduino "libraries" set-up as true link libraries. Setting up a scheme where each target platform (Uno, Duemil, the two Megas, etc.) would have it's own .a file for each library. Each .a file would get built by make on demand if it id didn't already exist the first time it was needed (or rebuilt if any of the dependencies had been changed, of course).  So you could have

$(ARDUINO_DIR)/libraries/DS1307/uno/DS1307.a

and

$(ARDUINO_DIR)/libraries/DS1307/mega2560/DS1307.a

for example.

Seems sensible and obvious. (Unless anyone can think of a reason this won't work.)

Martin Oldfield's scheme in v0.8 builds the .o files in $(SKETCH_DIR)/build-cli/libs/DS1307/DS1307.o for example. Which is consistent with the Arduino IDE approach of compiling Arduino "library" files as if they are just additional source files belonging  to the current project, so I suppose that's why he's gone that way.

Nothing I can't live with for the time being, though.

It's fast. It's simple. It's transparent. And eliminates an entire class of bugs (usually just called "Processing", LOL). Really, what's not to like?

I'm a _much_ happier Arduino camper than I was about 24 hours ago!  :-)
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: westfw on Jul 02, 2012, 07:09 am
Quote
get the Arduino "libraries" set-up as true link libraries.

Is there a reason to do this in addition to the "-ffunction-sections"  "--gc-sections" options?

Do set the Arduino IDE to "verbose" mode and watch what it does before you go off trying to create a makefile from scratch...
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 02, 2012, 08:10 am

Quote
get the Arduino "libraries" set-up as true link libraries.

Is there a reason to do this in addition to the "-ffunction-sections"  "--gc-sections" options?


Mostly just to avoid the unnecessary recompilations, I guess. The "make" way of doing things is to rationally manage a build so things don't get recompiled unless they actually need to be, after all. So a little bit of a speed of build benefit, and probably a bit more satisfaction that things are being done "properly", if you will. There is some happiness to be had in having a sharp set of tools. But as you point out, perhaps no difference in terms of what the .hex ultimately looks like.


Do set the Arduino IDE to "verbose" mode and watch what it does before you go off trying to create a makefile from scratch...


Good advice -- I will certainly do this if I proceed. But in the near-term, I'll go with the system as it is, in order to get a bit of confidence it has indeed covered all the bases as well as it appears.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 04:55 am
For completeness, I should post a link to Martin Oldfield's web page:

https://github.com/mjoldfield/Arduino-Makefile

there are also appear to be a few forks from this to check out if interested e.g.

https://github.com/peplin/arduino.mk

as well as the one by Sudar mentioned above (neither of which I've played with yet.)

Latest cool thing I've discovered: If you want to use your own main.cpp rather than the default "core" version supplied in the IDE distro, easy peasy: Just create a source file in your sketch directory named main.cpp, and will be used to build main.o rather than the use the "core" source file. If main.cpp isn't found, the default one will be used instead. Not sure if this is intended behaviour or just a fortunate side-effect of the order in which the make recipes are executed, but cool nonetheless.

The only thing that has to be in your main.cpp if you want to use the wiring libs is #include <WProgram.h> (or <Arduino.h>), and call init() before calling setup() (or however you decide to organise your main() function.)

Of course, also include any prototypes if you are calling anything beside init(), setup() and loop(), e,g.:

Code: [Select]

#include <WProgram.h>

void setup2();

int main(void)
{
init();

setup2();
   
for (;;)
loop();
       
return 0;
}


(this was my test alternative main.cpp to see if it all works as expected.)

Anyway, this is all too great. Using my own editor and the makefiles, there isn't any need to even fire up the IDE at all (unless I feel the need for a quick and dirty in "Processing" I suppose. But since I usually have emacs running anyway, I can't see this happening very often, somehow...)

Woot!
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: westfw on Jul 03, 2012, 06:15 am
Quote
unless I feel the need for a quick and dirty in "Processing" I suppose

Just a nit: it's NOT "Processing."  Processing is a desktop programming environment based on Java in about the same way that the Arduino IDE is based on C++ (http://processing.org )
I don't think that there's a name for the IDE pre-processing that occurs on your .ino files and whatnot.  "Arduino IDE preprocessing" is probably the most accurate phrase...
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 06:25 am

Quote
unless I feel the need for a quick and dirty in "Processing" I suppose

Just a nit: it's NOT "Processing."  Processing is a desktop programming environment based on Java in about the same way that the Arduino IDE is based on C++ (http://processing.org )
I don't think that there's a name for the IDE pre-processing that occurs on your .ino files and whatnot.  "Arduino IDE preprocessing" is probably the most accurate phrase...



Thanks for the clarification. I thought Arduino had simply adopted "Processing" as their beginners' language for the IDE. Is this just me or this a common misperception? I don't honestly even remember hearing about either "Processing" or "Wiring" before becoming acquainted with Arduinos.

Unfortunately, "Arduino IDE preprocessing" is a bit of a mouthful. So I think I shall adopt the Victorian approach and simply declare "and from this day forth we shall never speak of this again".

Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: westfw on Jul 03, 2012, 07:32 am
It's a common misconception that "Arduino" is some kind of specialized language, rather than pretty straight C++ with a bit of pre-processing applied.  About all that really happens is automatic prototype generation and concatenation of the .ino files (rather than compiling each separately.)  (Unfortunately, the IDE is a little clumsy about exactly where those prototypes should be added, leading to annoying incompatibilities with some common C preprocessor-directive styles...)
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 08:01 am

It's a common misconception that "Arduino" is some kind of specialized language, rather than pretty straight C++ with a bit of pre-processing applied.  About all that really happens is automatic prototype generation and concatenation of the .ino files (rather than compiling each separately.)  (Unfortunately, the IDE is a little clumsy about exactly where those prototypes should be added, leading to annoying incompatibilities with some common C preprocessor-directive styles...)


It also attempts to manage header files for you, which is where I suspect most of the bugs come from.

I was also under the impression it was pretty heavily subsetted version of C++ down to not much more than straight C (which of course id a proper subset of C++). And of course, a "sketch" being limited to one source file. Indeed, about the only hint that C++ is actually the underlying compiler rather than C is that class method reference constructions such as Serial.begin() are allowed. I don't think you can actually define classes within the that-which-must-not-be-named though (although I've not tried, I just went to a creating "library" *.cpp file for that whenever needed.)

FWIW, I've never really been convinced that automatically generating prototypes or managing header file inclusions are that much of a big deal that would constitute much of a benefit to novice programmers. Maybe the strict subsetting is helpful, I don't know. Is it useful or counterproductive to force beginners to put everything into one pde or ino file? Again, I don't know.

In any case, I'm just glad to have it behind me, whatever the hell that was.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: nickgammon on Jul 03, 2012, 08:11 am
I mention in the sticky here (http://arduino.cc/forum/index.php/topic,97455.0.html) that the underlying language is C++. I think it's an important point to make (which is why it is point #1).

With the exception of exceptions (lol) most other things can be done, and done easily. For example, you can use the STL if you download Andy Brown's port of it (The Standard Template Library (STL) for AVR with C++ streams  (http://andybrown.me.uk/ws/2011/01/15/the-standard-template-library-stl-for-avr-with-c-streams/)).

If you can use that (which relies heavily on templates) you can do pretty-much anything.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 08:29 am

you can do pretty-much anything.


So you can define your own classes in a pde/ino file? As I say, I never tried, but I was under the impression you had to create a seperate .cpp file to do that.

Whatever it is, it really aint C++ if you can't define classes.

Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: nickgammon on Jul 03, 2012, 08:38 am
Well you can, as this illustrates:

Code: [Select]
typedef class
  {
  public:
    int bar;
  } tFoo;

tFoo a;

void handler ()
  {
  a.bar = 42; 
  }
 
void setup ()
{
  handler ();
}

void loop () {}


However there is a problem if you try to pass your own defined class to a function, because of the automatic function prototype generation, which puts the prototypes before where the type is defined.

I just raised a bug report about that a few minutes ago:

http://code.google.com/p/arduino/issues/detail?id=973

On the bright side, if I may say without sounding pompous, most people who define a class in C++ would put it into its own .h files. That is really where class definitions belong. Then they can be shared between multiple .cpp files. You can do that under the Arduino IDE (making a new tab), it compiles correctly, and is easy to work with.

Quote
Whatever it is, it really aint C++ if you can't define classes.


Yep, you can define classes.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 08:45 am
OK, I suppose I should have said define *and* use. LOL.

Sort of like a write-only memory device. ;-)

In any case, I'm glad to be in C++. No questions of "pretty much" or "bug reports".


Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 08:55 am

most people who define a class in C++ would put it into its own .h files. That is really where class definitions belong.


I would disagree. Or say that if indeed most people are doing it this way, they are doing it wrong. You put the declarations (basically, prototypes for the class methods) into a .h file, but the the source for the methods is in a .cpp file. You definitely *don't* want this included in every file that simply want to use a class!
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: nickgammon on Jul 03, 2012, 09:07 am

OK, I suppose I should have said define *and* use. LOL.


My example used the class. It compiles.

Quote
You put the declarations (basically, prototypes for the class methods) into a .h file, but the the source for the methods is in a .cpp file. You definitely *don't* want this included in every file that simply want to use a class!


I don't get your point here. You put the definition in a .h file. You can put the implementation into the .ino file, or make another .cpp tab for it. Probably neater to put it into a separate file. So say you want implement class foo, you make foo.h and foo.cpp. What is the major problem with that?

Quote
In any case, I'm glad to be in C++. No questions of "pretty much" or "bug reports".


Right. So C++ has no bug reports?


But I've come across weirder and harder to understand (and therefore workaround) bugs -- most recently I had some code that would throw up spurious errors (some name collision of some sort going on under the hood, I suspect), but _would_ compile without error when the offending code was placed in a seperate .h file and then inlined in the sketch in the same place via a #include directive.

That just about did it for me.


Perhaps you should calm down and post the code in question, and try to understand the techniques required to work around any issues you have. The IDE has some idiosyncrasies, designed to make things easier for beginners. Now you can throw out the whole lot, and move to a pure avr-gcc environment, or keep the friendliness of the IDE and develop strategies for avoiding the issues that friendliness causes.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: nickgammon on Jul 03, 2012, 09:10 am
Let me put it to you like this. If you move away from the IDE, perhaps use a custom Make file, or use the Atmel Studio itself, you will no doubt eliminate some of the things that are bugging you. But, and this is important, you will replace them with other things that will annoy you. There is no perfect development environment.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 09:54 am

But, and this is important, you will replace them with other things that will annoy you. There is no perfect development environment.


No, of course not. But as there are degrees of evil, there are degrees of imperfection.

I probably should say, just to establish a bit of context, that I've dealt with many, many different development environments over the years, ranging over all sorts of hardware. The Arduino IDE was in no way the worst I've dealt wth. But very far from the best as well. (In fact, to be honest, it really does rank down there in the suckiness stakes. But I have dealt with worse.)

On a more positive note, there are many things apart from the IDE that I like about the Arduinos very much. I *love* the bootloader and the plug-and-play USB programming model. (I grew up with the old school development boards that you spent about a week getting set-up to get to "hello world" with. Intel, then Motorola, then MIPs.) I like the Atmel mega chips. I *love* the wealth of 3rd party libraries that means that just about any device you can think of interfacing with an Arduino probably has at least some basic code already available for it.

Which is why, overall, I am an Arduino fan.

But I am not a programming novice. I've been programming in C before there was even ANSI C, much less C++. Arduinos are for beginners. Fine. But I see no reason they have to be *only* for beginners.

So no, I don't expect "perfect" in a development environment. But emacs and gcc and make is pretty damned good, IMHO -- at least it's always something I've gotten along with. That's not to say "perfect". But, as they say, even the Sun has its spots. :-)

I'm glad there are options for those with differing tastes. Perhaps we can agree to agree on this.

Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: MichaelMeissner on Jul 03, 2012, 03:09 pm

But I am not a programming novice. I've been programming in C before there was even ANSI C, much less C++. Arduinos are for beginners. Fine. But I see no reason they have to be *only* for beginners.

Speaking as somebody who has used C since 1977 or 1978 and was on the original ANSI X3J11 C standards committee, I tend to agree with you in terms of the IDE trying to simplify and protect users.  But as you point out not everybody who programs Arduinos needs the training wheels.  In terms of development environments, I tend to be old school, and still haven't warmed to any graphical IDE, and only use it as a compilation unit and bootloader.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 05:07 pm

and try to understand the techniques required to work around any issues you have.


Found the definitive workaround, as discussed earlier in the thread.

You find the IDE "friendly". I find it a buggy PITA. Let's agree to disagree. And be happy for each other that you can use the IDE, and I can choose not to.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 03, 2012, 05:10 pm

In terms of development environments, I tend to be old school, and still haven't warmed to any graphical IDE, and only use it as a compilation unit and bootloader.


I suspect our tastes are similar, and I'm now very happy to report that I now don't even require the IDE for compilation or bootloading. Recommended.

BTW, good job on ANSI C.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pjrc on Jul 04, 2012, 04:36 pm

Mostly just to avoid the unnecessary recompilations, I guess. The "make" way of doing things is to rationally manage a build so things don't get recompiled unless they actually need to be, after all. So a little bit of a speed of build benefit, and probably a bit more satisfaction that things are being done "properly", if you will.


Did you realize Arduino 1.0.1 does this?  Seriously, it does.  I know....

I developed the code, originally in November 2010, as part of a patch on issue 393.  It was resubmitted 1 year later as issue 638.

http://code.google.com/p/arduino/issues/detail?id=638

Last December it was committed into Arduino's github repository, and eventually released as part of Arduino 1.0.1.

I'm sure that does nothing to sway you to prefer the IDE over vim & a makefile... but when talking about ways command-line based make is superior to the Arduino IDE, avoiding recompiling files based on timestamps and dependency analysis has been part of Teensyduino for 1.5 years and now, as of 1.0.1, is officially a feature of the Arduino IDE.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pjrc on Jul 04, 2012, 04:49 pm
You can bypass the IDE's preprocessing by placing all your code in .cpp and .h files, instead of .ino or .pde.  You still need to have at least 1 .ino or .pde file which has the same name as the directory it's in.  But that file can be empty.

The Arduino IDE supports .cpp, .c, and .h files within your sketch.  They are NOT preprocessed in any way.  If you want to use the normal Arduino functions, you must add the #include <Arduino.h> line in each .cpp file.

Within those .cpp files, you can define classes and use templates.  You can also use them in .ino files, but there's a pretty good chance the IDE's not-too-smart preprocessing step will attempt to extract global-scope function prototypes.  But that preprocessing stuff is NEVER used on .cpp files.

Likewise, the .cpp files are NOT parsed for includes to infer which libraries to link into your code.  So in that empty .ino file, you can place #include <Library.h> lines, merely as directives to the IDE's build system to tell it which libraries to link.  Then you can include only the library headers needed in specific .cpp files which call those library classes.... so your other .cpp files remain free of namespace clutter.

Very few people seem to know this feature exists, but indeed the Arduino IDE does support real C++ compilation without troublesome preprocessing.  You merely need to add the .cpp files to the directory, and then do your coding in those .cpp files instead of .ino files.

Of course, there are a couple C++ things you don't get, not because of the IDE, but due to limitations in the underlying AVR toolchain.  The avr-gcc compiler does not support exceptions, and avr-libc does not provide the C++ stdlib classes & functions.

But you can bypass the preprocessing step.  It's as easy as just creating .cpp files in the sketch directory!
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pico on Jul 04, 2012, 06:21 pm

Very few people seem to know this feature exists, but indeed the Arduino IDE does support real C++ compilation without troublesome preprocessing.  You merely need to add the .cpp files to the directory, and then do your coding in those .cpp files instead of .ino files.


Excellent info, thanks Paul for the tips and insights. My cup runneth over with options... :-)

Choice is good. And Arduino development with the option to bypass the "helpful" preprocessor is better.

Hopefully a few more people will now be alerted to these basic choices available. I'm sure I can't be the only one glad to get the "sketch" preprocessing out of the picture! :-)

Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: nickgammon on Jul 05, 2012, 12:12 am

You can bypass the IDE's preprocessing by placing all your code in .cpp and .h files, instead of .ino or .pde.  You still need to have at least 1 .ino or .pde file which has the same name as the directory it's in.  But that file can be empty.

The Arduino IDE supports .cpp, .c, and .h files within your sketch.  They are NOT preprocessed in any way.  If you want to use the normal Arduino functions, you must add the #include <Arduino.h> line in each .cpp file.


I mentioned that in reply #1, but thanks Paul for putting it more forcefully.

Also thanks for the tip about the way that unnecessary file are not recompiled every time. The IDE keeps getting better!
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: bperrybap on Jul 19, 2012, 12:11 am


Mostly just to avoid the unnecessary recompilations, I guess. The "make" way of doing things is to rationally manage a build so things don't get recompiled unless they actually need to be, after all. So a little bit of a speed of build benefit, and probably a bit more satisfaction that things are being done "properly", if you will.


Did you realize Arduino 1.0.1 does this?  Seriously, it does.  I know....

I developed the code, originally in November 2010, as part of a patch on issue 393.  It was resubmitted 1 year later as issue 638.

http://code.google.com/p/arduino/issues/detail?id=638

Last December it was committed into Arduino's github repository, and eventually released as part of Arduino 1.0.1.

I'm sure that does nothing to sway you to prefer the IDE over vim & a makefile... but when talking about ways command-line based make is superior to the Arduino IDE, avoiding recompiling files based on timestamps and dependency analysis has been part of Teensyduino for 1.5 years and now, as of 1.0.1, is officially a feature of the Arduino IDE.


I'll agree that the update to the IDE improves the Arduino build process,
But the IDE even with this improvement still doesn't completely avoid doing unnecessary things or do what can be done with Makefiles.
Yes the update can prevent rebuilding of certain files for a given sketch under certain circumstances,
but since the IDE doesn't build core or Arduino library archives and store them independently of IDE sketch build area,
the IDE must rebuild library archives when say changing board types or restarting the IDE.
Because of the build methodology, the IDE also doesn't (can't) remember the compiled objects across instances of the IDE
even for the same sketch.
i.e. bring up IDE, build a sketch, exit IDE, bring up IDE for the same sketch - EVERYTHING rebuilds.
Even in the cases where the IDE is smart enough to not to re-compile core objects, it still
re-archives them into a local core library.

Overall, it is a build methodology problem. The IDE falls down in certain advanced capabilities because of
the methodology used and it is very difficult if not impossible to push the IDE to be "smarter" do things
differently without having to actually modify the IDE JAVA code.

With makefiles it is possible to do things in a more traditional way like
create 1 core library and 1 set of true libraries for each board type
and with proper dependencies never have to rebuild them again no matter what sketch uses them
or across sketch builds unless there is a change to the actual library code.

Consider an IDE like Atmel's AVR studio. It's build methodology was to use build on top of Makefiles so
that it can do things like allow you to  override everything and use your own makefiles.

What annoys me the most about the IDE is that while it works quite well for certain work flows
and for novices, some of its implementation decisions - most of which would not have affected novices work flow
or impact their ease of use, get in the way of or even prevent doing certain more advanced things.

An Arduino library being able to call another Arduino library is very problematic with the current IDE build methodology.
Yeah, there is a hack to allow it to work today, but it involves adding header files to the actual user sketch - which is a
kludge since the sketch really doesn't care about such sub library dependencies.
Another, being able to source level debug using other Atmel tools.
Those types of problems can go away if a slightly different build methodology were used.

So for me, the building/re-building of modules unnecessarily isn't my biggest concern, what concerns me more is that
certain things like conditional compilation, (sketch being able to control compilation options of libraries), libraries being
able to call other libraries, library code being able to detect and conditionally change based on board type
(not processor type, or having to poke around for "magic" defines), being able to source level debug are the killers.
Heck even the ARDUINO define for the core code revision is not available in a header file as the build methodology
assumed things were only going to be built with the IDE.
It would be nice if the IDE developers would at least consider the ability to build makefiles around their core libraries
for the more advanced uses that want to use Makefiles so that they can do things like source level debugging
as some of the way they are doing things make it more difficult for building Arduino sketches with Makefiles.


--- bill
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: pjrc on Jul 19, 2012, 05:53 pm
Quote
Yes the update can prevent rebuilding of certain files for a given sketch under certain circumstances,
but since the IDE doesn't build core or Arduino library archives and store them independently of IDE sketch build area,
the IDE must rebuild library archives when say changing board types or restarting the IDE.


Make, ant, or other build tools can't reuse code between board changes either.  The Arduino core and most libraries contain MANY #ifdef checks on the processor and clock frequency.  When you change to a different board, all that code simply must be recompiled.

Compiled code is indeed not saved between IDE sessions.  Is that really a big deal?  Really?

Long ago, Arduino compiled code to an "applet" subdirectory within the sketch.  The change to using a temporary directory was primarily motivated by Arduino's usage in schools with tightly locked down systems.  One of the few places you can be absolutely certain is writable under even the most strict security is a temporary directory.

Quote
An Arduino library being able to call another Arduino library is very problematic with the current IDE build methodology.  Yeah, there is a hack to allow it to work today, but it involves adding header files to the actual user sketch


This is issue #236.  I will fix it.  I would have fixed it about 6 weeks ago, had David not temporarily changed his mind on this issue.

Quote
... what concerns me more is that certain things like conditional compilation, (sketch being able to control compilation options of libraries), libraries being able to call other libraries, library code being able to detect and conditionally change based on board type (not processor type, or having to poke around for "magic" defines), being able to source level debug are the killers.


Are you sure that's it?  I mean, really, if these things were all fixed, I have a strong feeling you and many, many others who don't like the IDE still wouldn't like it.

Except for source level debugging, these are all pretty simple things.  Already it is possible for sketches to control library options using C++ templates.  The USB Host Shield library is one that leverages this to great effect.  Others do so much less elegantly with headers to inline some code (my own Encoder library would be a good example of the dirty way).  A standard way to detect the actual board has been an open issue for a long time, and one I'd personally like to patch.  So far, the Arduino developers have shown little interest, but I'm patient and persistent.  As more boards come out, it will become more compelling.  Eventually I will fix that.  Only days ago David (finally) relented and will accept a fix for issue 236, and I will write it in several weeks when I'm done with my current project.

I'm sure there are LOTS of other little things you and many others will name as Arduino IDE deficiencies once all this stuff is fixed.  The simple fact is Arduino is designed to be simple and NOT have lots of advanced options.  There are always going to be lots of things present in AVR/Atmel Studio, Eclipse and Visual Studio that won't be in Arduino.

But I do agree, source level debugging is the big issue.  Of course, it's an absolutely non-starter with AVR.  Atmel absolutely will not provide the debug specs for their AVR chips.  Nobody, other than Atmel, can reasonably work on this (short of impractical reverse engineering) as long as AVR is the platform.
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: bperrybap on Jul 19, 2012, 11:25 pm

Quote
Yes the update can prevent rebuilding of certain files for a given sketch under certain circumstances,
but since the IDE doesn't build core or Arduino library archives and store them independently of IDE sketch build area,
the IDE must rebuild library archives when say changing board types or restarting the IDE.


Make, ant, or other build tools can't reuse code between board changes either.  The Arduino core and most libraries contain MANY #ifdef checks on the processor and clock frequency.  When you change to a different board, all that code simply must be recompiled.

Depends on how it is implemented. If the build system creates a library archive for each board type,
the library code for that board type is only ever built once or whenever the library code changes.

There could also be an implementation where Arduino libraries are "installed".
Where at install time, the library code is then built for all board types and archived into the appropriate library archives.

The Arduino IDE could be updated with all the "smarts" to do this kind of stuff as well
so it is more of a build methodology issue than a specific tool issue.

Quote

Compiled code is indeed not saved between IDE sessions.  Is that really a big deal?  Really?

I agree that it is not that big of a deal. I even stated as much earlier. I was simply pointing out (in a long winded fashion)
that the IDE even with the additional smarts is still not quite fully doing what Pico was wanting/asking for
as even with its additional smarts it still sometimes re-compiling modules that have not changed.
(For the record, I didn't think the dumb way of building everything every time was that bad)

But in the bigger picture I wish if the IDE were not going to really do things "right" with dependencies
then I kind of wish it would even be simpler/dumber and simply glomb everything together into one HUGE compile unit rather
than to try to pretend to deal with things like library archives.
Compile times wouldn't be any longer than what was being done before the "smart build" updates, and
if everything were glombed together there are other compiler optimizations that could be done.
For example, it would be possible to crush down things like digitalWrite() into single instructions
as the compiler could now have visibility into all the mapping tables. This would allow getting the optimized
AVR bit set/clr instructions when using constants without having to resort to using separate matching macros to map things
when possible.

Quote

Long ago, Arduino compiled code to an "applet" subdirectory within the sketch.  The change to using a temporary directory was primarily motivated by Arduino's usage in schools with tightly locked down systems.  One of the few places you can be absolutely certain is writable under even the most strict security is a temporary directory.

I'm familiar with that and understand why they did what they did.
I don't necessarily disagree with it given the state of the rest of the IDE build process, but
I believe that there are some other alternatives like creating build areas under users sketchbook area
that offer the potential for more advanced/smarter capabilities.

Quote
An Arduino library being able to call another Arduino library is very problematic with the current IDE build methodology.  Yeah, there is a hack to allow it to work today, but it involves adding header files to the actual user sketch


This is issue #236.  I will fix it.  I would have fixed it about 6 weeks ago, had David not temporarily changed his mind on this issue.
Quote

Quote
... what concerns me more is that certain things like conditional compilation, (sketch being able to control compilation options of libraries), libraries being able to call other libraries, library code being able to detect and conditionally change based on board type (not processor type, or having to poke around for "magic" defines), being able to source level debug are the killers.

Are you sure that's it?  I mean, really, if these things were all fixed, I have a strong feeling you and many, many others who don't like the IDE still wouldn't like it.
***
I'm sure there are LOTS of other little things you and many others will name as Arduino IDE deficiencies once all this stuff is fixed.  The simple fact is Arduino is designed to be simple and NOT have lots of advanced options.  There are always going to be lots of things present in AVR/Atmel Studio, Eclipse and Visual Studio that won't be in Arduino



But I don't believe that
"Simple" has to preclude eliminating advanced capabilities.

I'm not a whiner. What I do tend complain about is when ease of use simplifications come at the expense of restricting
or even precluding more advanced functional capabilities,
especially when it is do to implementation/design decisions that really don't impact the simpler
uses or cost additional resources to implement.
For example, in some situations you can go down path A or B and neither affects high level usability
or ease of use but perhaps one make its easier or harder to do more advanced things.
This is some of the things I've seen in Arduino not just in the IDE but also in some of the core libraries.
I draw a clear distinction between the IDE and the Arduino software environment.
From my outsiders point of view, the Arduino teams vision doesn't seem to include much thought about this distinction.
It seems to be very much everything "Arduino" and the IDE and the s/w environment (core code and libraries)
are all seen as one.
So there doesn't seem to be much thought given to making the s/w environment more friendly to more advanced users
or users that wish to do Arduino development but perhaps not with the IDE.

In other words take a look at issue #236. It is a great example that shows how things are inter related.
If there were a board type Arduino library archive created for all the "installed" libraries
there would be no issue trying to figure out which libraries to link in. You could simply
add the top level libraries directories to the include path which would allow each library to locate other libraries
header files by using #include <libname/libheader.h>,
then link against the Arduino library archive for the currently selected board type.
So things come back full circle.
If the build methodology created board type Arduino library archives, not only could unnecessary
compiles be avoided, but it also makes things like linking in library code and handling sub library
dependencies easier.

Not that it has to be done that way, but just an example how things are inter related and how
things cascade by certain decisions in the build methodology.

For me, is isn't so much what the IDE does or doesn't do. It is more the build methodology and
what it prevents or makes difficult, whether that is using the IDE or not using the IDE.
So for example, I think all the talk about building/rebuilding objects is pure noise.
That kind of stuff is not a functional issue. It does not prevent or preclude anything from being done.
(Makefiles can be written to be just as dumb)
However, look at a few of some of items the advanced users are wanting:
- libraries calling other libraries
- ability to easily identify board type at compile time
- ability to change/modify compile/linker options
- ability to set compiler defines on a per board type basis.
- ability to do source level debugging
- enable floating point libraries, like printf()
- better/new define names for pins to avoid having to detect and adjust code based on board types.

Those are functional issues. And currently the only way to make many of these work
is to either to modify the IDE or jump to something like your own Makefiles.

MPIDE handles several of these things much better in this area than the Arduino IDE as you can, if you choose, have more control
over the compile & link parameters all the way down to the board type level.

So overall I do wish there was a bit more thought going on about the Arduino s/w development
build process to ensure that things are easier for building more complex Arduino projects.
And just because the project is more complex doesn't mean that this advanced capability is
directly exposed to the end user or that it is more complex for the novice user
to build it or use it.



Quote

But I do agree, source level debugging is the big issue.  Of course, it's an absolutely non-starter with AVR.  Atmel absolutely will not provide the debug specs for their AVR chips.  Nobody, other than Atmel, can reasonably work on this (short of impractical reverse engineering) as long as AVR is the platform.

Not necessarily. You can do it purely in software without having to deal with any of their
debugging hardware or interfaces. I know how to do source level debugging using only a serial port with gdb on the AVR.
It does require stealing a hardware pin to steal an interrupt,
and a very small (hundred or so byte support code monitor) but it can be done.

--- bill
Title: Re: Option to bypass "Processing -> C++" pre-processor
Post by: ardnut on Aug 11, 2012, 06:20 pm
Quote
Not necessarily. You can do it purely in software without having to deal with any of their
debugging hardware or interfaces. I know how to do source level debugging using only a serial port with gdb on the AVR.
It does require stealing a hardware pin to steal an interrupt,
and a very small (hundred or so byte support code monitor) but it can be done.


Useful. Got any more detail on that?