Arduino Forum

Development => Suggestions for the Arduino Project => Topic started by: pjrc on Sep 13, 2011, 08:44 pm

Title: Compile Speed - testing & feedback needed!
Post by: pjrc on Sep 13, 2011, 08:44 pm
I wrote code that cuts several seconds off the normal compile/upload time.  Here's a little video I made yesterday to demonstrate.

http://www.youtube.com/watch?v=2dBF1ypQupM

This works much like Make, Ant and other build systems, by reusing the previously compiled files from the last time.  I actually wrote this almost a year ago, but it was considered too risky to use in Arduino until more people have tested it and confirmed it really works.  So the code sat unused and forgotten for almost a year!

Well, now that Arduino 1.0 is here (or will be in a few days), I'm hoping to revive this and hopefully get enough people to test so it can someday become part of Arduino.

Here's the page with the installer that adds it. 

http://www.pjrc.com/teensy/td_download.html

If you're using a non-Teensy board, you'll also have to edit boards.txt.  The line that enables it looks like this:

Code: [Select]
uno.build.dependency=true

You should notice a substantial speedup compiling the same sketch.  Hopefully when you switch sketches, board types, or have multiple windows open at once... stale code from one should not be used again in another.

If you give this a try, working or not, please take a moment to comment on issue 638.

http://code.google.com/p/arduino/issues/detail?id=638

This feature has sat unused since November 2010, and it may never get much use beyond Teensy boards without your help!
Title: Re: Compile Speed - testing & feedback needed!
Post by: BAXX on Sep 13, 2011, 09:33 pm
Looks great, I will test is for sure ;)
thanx
Title: Re: Compile Speed - testing & feedback needed!
Post by: CapnBry on Sep 19, 2011, 09:53 pm
I didn't want to add just another "Me too" to your ticket, but using Arduino-0022, Arduino 1.0 Beta 3, and Arduino 1.0 Beta 4: I've installed teesyduino into each of those directories and, while I do get the teensy additions showing in the IDE, modifying boards.txt (and restarting) does not change the compliation. It still recompiles all files.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Sep 19, 2011, 10:27 pm
Without more info, I can't know why it's recompiling everything.

If you're willing to try more, here's a few suggestions.

1: Can you post the verbose messages.  On 1.0-beta4, enable in the preferences.  On 0022, hold the shift key.  The speedup only applies to "upload", not "verify".

2: Can you post your boards.txt file?  There's several boards in the file, so make sure you're added this line for the board you're using.

3: Can you tell me which operating system and which board you're using?  I have most combinations here.  While it should work on all, I've only tested a small set of all the possible permutations (eg, only Uno and Teensy 2.0 on Windows, etc)

4: Could you reinstall Teensyduino 0.98?  If you have 0.97 or earlier, there was a bug parsing the dependency files.  It turns out the Windows and Mac versions have extra white space which isn't present on the Linux version.
Title: Re: Compile Speed - testing & feedback needed!
Post by: CapnBry on Sep 19, 2011, 11:09 pm

1: Can you post the verbose messages.  On 1.0-beta4, enable in the preferences.  On 0022, hold the shift key.  The speedup only applies to "upload", not "verify".

Oh that would do it!  Yeah I was only doing the compile / shift compile. With upload you're right it works great!  Will post to your ticket with my results.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Sep 20, 2011, 01:23 am
Oppps, guess I could have mentioned it only speeds up "upload"  ;)

I actually spent a lot of time editing that video down to only 2.5 minutes.  Maybe I cut too much?

Glad it's working.  Hopefully anyone else who tries it will see this and know to use Upload instead of Verify.
Title: Re: Compile Speed - testing & feedback needed!
Post by: CapnBry on Sep 20, 2011, 03:20 am
I thought the video was very professional actually! Arduino videos are synonymous for me with "Ummm hey guys, this ummm an arduino" with the video being taken by a cellphone camera shakily shooting a monitor you can't read. Your video a welcome surprise-- brief, to the point, informative, and clear.

One of my projects is around 60 files and this is so nice to be able to compile it much faster. If I can ever get git to clone to my machine, I'd love to add an option to skip the verify step of the avrdude upload and get my 30KB compile + upload down to 15s total!

Another important thing for others to note: When you hit "Save" in the Arduino IDE, it cleans the build directory and removes all the pre-compiled files. This also prevents this optimization from being used.
Title: Re: Compile Speed - testing & feedback needed!
Post by: bill2009 on Sep 21, 2011, 01:13 am
So this would only speed up the part until the size of the module is displayed - right?
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Sep 21, 2011, 01:48 am
Yup, it only speeds up the compile part.

If you're using Teensy, the data transfer goes pretty fast, since it's full USB speed.  I recently discovered that Teensy can go even faster.  It turns out much of the time is wasted waiting for operating system scheduling.  I'm planning to overhaul the process to use overlapping async I/O to remove the dependency on operating system scheduling.  I like things to go fast!

The process for official Arduino boards isn't under my control.  In fact, with Teensyduino, I try very hard to make sure I don't accidentally alter anything about how non-Teensy boards work.  That's why you need to copy that line into boards.txt to enable this.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Sep 22, 2011, 06:24 pm
As a final followup, I prepared a patch and posted it to issue 638.

http://code.google.com/p/arduino/issues/detail?id=638

Due to the underwhelming response, I'm not planning to work further to contribute this feature to the larger Arduino community.  The source is there if anyone wants to use it, or continue to improve it and advocate its use.
Title: Re: Compile Speed - testing & feedback needed!
Post by: sssz3 on Nov 06, 2011, 06:59 pm
I gave it a try___ and it worked out for me... Thank you very much, I was so tired of the long building times.
In my case, building time (without upload) was reduced from 15 sec to 5 sec, if I only make a few changes in code, e.g. adjust some timings.

Now I will get Optiboot to speed up the upload, and voila, i will get the max out of my development time...

Thanky.... ;)


Title: Re: Compile Speed - testing & feedback needed!
Post by: CrossRoads on Nov 11, 2011, 05:57 pm
Paul,
Maybe with the release of Leonardo & ATMege324U4 with built in USB, this will come in handy as a tool.
I didn' really notice long compile times before, guess I am not writing long enough programs.

Robert
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Nov 12, 2011, 07:35 pm
Yes, the upload to Arduino is slow due to the limited serial speed.  It's hard to notice how long the compile takes, because the slow upload takes much longer.

I developed this compile speedup for Teensyduino, which uses the 32u4 chip (and has been Arduino compatible for nearly 2 years).  If you've never tried a Teensy, you'll probably be surprised how fast uploading code is.  With Teensy's fast data transfer speed, recompiling everything became most of the wait, but this change eliminates most of that wait too.

Here's a video I recently uploaded.  Check out the upload speed.  It's at approx 0:42 in the video.

http://www.youtube.com/watch?v=BHQPqQ_5ulc

Did you actually try this speedup?  How much speedup difference does it make on your computer?
Title: Re: Compile Speed - testing & feedback needed!
Post by: CrossRoads on Nov 13, 2011, 01:31 am
I have not tried  it.  However, I am putting together a Duemilanove type board with the ATMega32U4, and I'm thinking the Teensyduino code could be good on that.

I haven't decided what to do with the extra 6 pins yet; I have going to a jumper block to allow I2C or A4/A5 to connect to the header, and put some parts in support the PE2 & PE6 pins. That leaves PC6 & PC7 still unconnected.

I have assigned the connections to the headers to maintain compatibility, this makes the layout work ok, not sure if the assignments make sense or how much pins_arduino.c will need to be revised.
Title: Re: Compile Speed - testing & feedback needed!
Post by: westfw on Nov 14, 2011, 06:19 am
How does the installer work?  Is it version independent, or only applicable to 0022?

I'm hoping that Arduino will eventually go to a make-based compilation, even if the default behavior is to recompile everything.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Nov 14, 2011, 04:04 pm
I have done everything I possibly can to get this improvement into the official Arduino, even to the point of making a video demonstration to solicit more beta testing, since the Arduino Team believed this feature might be too risky without more testing.

My basic feeling is it's nearly impossible to contribute anything to Arduino, other than specific bug fixes.  Even "patches welcome" doesn't mean contributed code will actually be used.  For example:

http://code.google.com/p/arduino/issues/detail?id=550

Here's another example, where a feature was requested (by the Arduino Team), and I went to the trouble to produce a patch, and screenshots.  They didn't like the syntax I chose, but indicated the feature itself was good.  So after some back-and-forth to clarify what syntax would be desired, I pretty much redesigned the entire thing and submitted a second patch.  It has sat unused for 1 YEAR.

http://code.google.com/p/arduino/issues/detail?id=257

I'm particularly bitter about the very substantial work I put into that redesign.  The original approach tried to limit the scope of changes within Arduino.  Since they haven't used it, I'm left with a larger patch set which I maintain separately for Teensyduino.  Never again will I rework a patch like that.

Quote
I'm hoping that Arduino will eventually go to a make-based compilation, even if the default behavior is to recompile everything.


Bill, you know a make-based build has been brought up over and over on the developer mail list.  The idea has been consistently rejected.

Given how hard it is to get any patch accepted (other than specific fixes to accepted bugs), even in cases like these where the idea was approved, even when "patches welcome" is said, obviously there's zero chance Arduino will ever use make, even if someone is foolish enough to try contributing that code.


The Teensyduino installer is my kludge to avoid completely forking Arduino.  In hindsight, forking would be easier.  My installer is NOT version independent.  With each new Arduino version, it needs to be updated.  It only works well because I put a huge amount of work into ongoing maintenance.  I'm a little crazy like that....

I keep hoping Arduino will be more open.  In hindsight, I regret putting so much work into sharing this compile speedup.  I've easily spent 10X more time trying to get Arduino to actually use it than I did writing it in the first place!
Title: Re: Compile Speed - testing & feedback needed!
Post by: westfw on Nov 14, 2011, 05:50 pm
There's "wiring" ...
I've been wondering whether they've been collecting frustrated IDE developers...
Title: Re: Compile Speed - testing & feedback needed!
Post by: razrburn on Dec 01, 2011, 12:17 am
This is making me happy that I have 3 teensy's rather a arduino, besides this modification there is so many other things that specifically make a teensy great... I'm really surprised to see so many people buying a arduino and trying to use it as a g-code interpreter for a cnc. Teensy' 2+ offers so much more program space and communication speed...

I'd like to thank paul for supporting open source. I've seen several attempts of him trying to contribute to different projects.
if arduino doesnt resolve this issue, i'd still like to say your hard work has been appreciated by at least 2 of your customers. if only the whole community could enjoy your fixes without such a long delay from the arduino team.

Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Dec 01, 2011, 04:30 am
Maybe now that Arduino 1.0 is finally released, hopefully they'll have more time to look at "post-1.0" contributions like this speedup?
Title: Re: Compile Speed - testing & feedback needed!
Post by: mellis on Dec 02, 2011, 03:36 pm
That's my hope.

Sorry, Paul, I know it's frustrating to submit patches that languish.  I just haven't had time to properly handle everything.  That was one of the reasons I wanted to get the 1.0 release out - so that I could go back and start integrating more of the patches and changes that have been accumulated.  I'll do my best to get them sorted soon.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Dec 02, 2011, 04:30 pm
Certainly there are much more urgent things than compile speedup.  Issue 527, for example.

It definitely is frustrating to see patches go unused.  I would contribute a LOT more if my existing patches were used, and Arduino in general had an openness towards contributions that was reflected in the actual usage of patches people submit.  I'm imagine quite a few other people, some of whom I know have utterly given up on contributing to Arduino.

On unused patches, I'd like to remind you about the one I submitted on issue 550.  You wrote "patches welcome" in October.  I submitted the patch 2 days later, and an improved patch after another 3 days (based on some user feedback from the first).  It has now sat for a full month.  Will it ever get used?

Especially when you say "patches welcome", but then you don't use them, that is really disheartening to contributors.

Another perfect example is "menu cleanup" (issue 257), which has sat unused for over 1 year!  I never expected my original patch to be used.  But then in November (1 year ago) you specifically asked me to completely rework the boards.txt syntax.  I did, and submitted a completely new patch within a week.  It was a substantial amount of work.  Why would you ask me to do all that work, but then never use it?

Still, I have at least a few more patches heading your way soon.  They're all little things recently discussed on the developer mail list.  I'm reluctant to submit any more "big" patches.
Title: Re: Compile Speed - testing & feedback needed!
Post by: JRobert on Dec 08, 2011, 07:29 pm
On the original topic (with no intent to discount the patches discussion):

I've been using teensy .99 for the "compile only what's needed" feature and it's been working wonderfully, with no issues, to reduce my debug turn-around times.  Thank you for the effort you put into it.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Dec 12, 2011, 03:18 pm
Glad to hear it's working and helping you.  :-)
Title: Re: Compile Speed - testing & feedback needed!
Post by: bamboosam on Mar 16, 2012, 05:42 am
Yeah your Teensy++ 2.0 kicks a$$! Its the greatest in between. Thanks for the great support also on it...
I just sort of found my self in a hard spot trying to develop my own custom micro with the at90usb1286 to fit my application. I know nothing on bootloaders.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Mar 16, 2012, 05:02 pm
Glad to hear Teensy is working out so well.   :)

This speedup was accepted into Arduino and will be in version 1.0.1.  If you're using Teensy, you already have it.  For normal Arduino, it can be tested now in the release candidates:

Mac OS X: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-macosx.zip
Windows: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-windows.zip
Linux (32-bit): http://files.arduino.cc/downloads/arduino-1.0.1-rc1-linux.tgz
Linux (64-bit): http://files.arduino.cc/downloads/arduino-1.0.1-rc1-linux64.tgz
Source: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-src.tar.gz

If anyone reading this wants an easy way to help Arduino development, installing and using the release candidates NOW and reporting any issues soon would really be good.


Title: Re: Compile Speed - testing & feedback needed!
Post by: bamboosam on Mar 17, 2012, 01:21 am
Will do
Title: Re: Compile Speed - testing & feedback needed!
Post by: bamboosam on Mar 17, 2012, 01:24 am
< Actually your Teensy is controlling the LED strip in my Avatar pic <
Title: Re: Compile Speed - testing & feedback needed!
Post by: MGeo on Jun 03, 2012, 06:42 pm

This speedup was accepted into Arduino and will be in version 1.0.1.  If you're using Teensy, you already have it.  For normal Arduino, it can be tested now in the release candidates:

Mac OS X: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-macosx.zip
Windows: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-windows.zip
Linux (32-bit): http://files.arduino.cc/downloads/arduino-1.0.1-rc1-linux.tgz
Linux (64-bit): http://files.arduino.cc/downloads/arduino-1.0.1-rc1-linux64.tgz
Source: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-src.tar.gz



I've updated to 1.0.1 to evaluate this feature.  I am not having luck finding the right setting to activate the incremental compile feature.  Tried adding  XXX.build.dependency=true to relevant boards in boards.txt but no luck.  Is activation of this feature documented anywhere?
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Jun 03, 2012, 08:20 pm
It's always enabled.

I used .build.dependency=true in Teensyduino's pre-1.0 patch, so it wouldn't change Arduino's behavior without explicitly being enabled.  Arduino 1.0.1 does not have that option, so it's always enabled.  There is no option to disable it.  But if you select another board from Tools > Boards, even temporarily, the next Verify or Upload will recompile everything.  So if you want to compare the speed, just change the board and then set it back, to do a test where all files are recompiled.

If File > Preferences you can activate verbose messages while compiling.  Those messages should indicate if the compiler is being run, or the already-compiled file is being reused.
Title: Re: Compile Speed - testing & feedback needed!
Post by: MGeo on Jun 03, 2012, 09:33 pm

If File > Preferences you can activate verbose messages while compiling.  Those messages should indicate if the compiler is being run, or the already-compiled file is being reused.


Thanks Paul.  I did a test build of the BareMinimum sketch build for an Uno board.  The two builds look identical to me.  I'm sure I'm doing something wrong, need to keep searching to figure out what.  Attached are the 1st and 2nd build files with verbose on.  Thanks for the reply.

George
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Jun 03, 2012, 09:40 pm
This feature works by comparing timestamps on the files.

Maybe the files in your arduino directory have timestamps in the future?  Perhaps if you extracted the zip file, then set your clock back?  For example, take a look at the file:

C:\Program Files\arduino-1.0.1\hardware\arduino\cores\arduino\wiring.c

If it's timestamp is in the future, then it will still be considered newer than the .o file.

The easiest solution might be to extract a fresh copy of Arduino 1.0.1.
Title: Re: Compile Speed - testing & feedback needed!
Post by: MGeo on Jun 04, 2012, 01:14 am
Thanks again for the input.

I checked the dates on my core folder, dated 5/21/2012.  My machine date is current at 03-Jun-2012.

I'll keep digging.  Thanks for the input.

George
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Jun 04, 2012, 02:01 pm
When you compile, all the  generated files are placed in a temporary directory.  With the verbose info, you can see the full pathname.  If you look in that directory, are there .o and .d files for each source?  Are the timestamps ok?

Could you please post one of the .d files, perhaps wiring_digital.d ?
Title: Re: Compile Speed - testing & feedback needed!
Post by: MGeo on Jun 05, 2012, 11:51 am
Here are temp folder window captures from 1st and 2nd build of BareMinimum.ino example sketch, along with wiring_digital.d of 2nd build:

I notice the paths listed in wiring_digital.d look a bit strange.  Under windows with a typical install located in "C:\Program Files\..." the path has a space.  I wonder if this could this be confusing the compare parsing algorithm?

Code: [Select]

\
C:\DOCUME~1\gsmall\LOCALS~1\Temp\build8448203176909195487.tmp\wiring_digital.c.o:  \
C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\wiring_digital.c \
 C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\/wiring_private.h \
 C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\/Arduino.h \
 C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\/binary.h \
 C:\Program\ Files\arduino-1.0.1\hardware\arduino\variants\standard/pins_arduino.h


Thanks for your help.

George
Title: Re: Compile Speed - testing & feedback needed!
Post by: MGeo on Jun 10, 2012, 11:44 am
1st and 2nd run of wiring_digital.c.d from temp build folder attached below.

All,

Has anyone else had this new for release 1.0.1 feature work properly?  If so, could you share your .d file.

Thanks,
George
Title: Re: Compile Speed - testing & feedback needed!
Post by: WizenedEE on Jun 12, 2012, 02:51 am
I have the git version of arduino, and the feature has worked properly for me for quite some time (months). Here's wiring.c.d after hitting verify twice
Title: Re: Compile Speed - testing & feedback needed!
Post by: MGeo on Jun 17, 2012, 01:05 pm
Thanks for posting.  Dependencies look similar to mine.  Looks like they are not supposed to change 1st run to subsequent runs.  Guess that makes sense.  The messed up path showns in the .d files on my Win 7 machine looks like a clue.  Hmmm...

Code: [Select]
\
C:\DOCUME~1\gsmall\LOCALS~1\Temp\build3787739127124592615.tmp\wiring_digital.c.o:  \
C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\wiring_digital.c \
  C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\/wiring_private.h \
  C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\/Arduino.h \
  C:\Program\ Files\arduino-1.0.1\hardware\arduino\cores\arduino\/binary.h \
  C:\Program\ Files\arduino-1.0.1\hardware\arduino\variants\standard/pins_arduino.h
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Jun 17, 2012, 04:04 pm
If you move your arduino-1.0.1 from C:\Program Files to just C:\ (at least temporarily for the sake of testing), does that make this magically start working?

FYI: lately I haven't had any time for Arduino hacking... 2 huge project deadlines, one recently passed, the other is a huge interactive fire art installation that shows next weekend.  I plan to look into this issue in early July.  If you could confirm (or refute) that moving the just C:\ fixes the problem before then, please do.
Title: Re: Compile Speed - testing & feedback needed!
Post by: timkay on Mar 16, 2014, 04:02 am
It's been 2 years since this issue was discussed but never resolved.

I am confirming that, YES, moving the Arduino folder from C:\Program Files to just C:\ resolves this issue. I see a speed up of at least 8X using 1.05-r2.
Title: Re: Compile Speed - testing & feedback needed!
Post by: pjrc on Mar 16, 2014, 07:01 am
Oh, yeah, I fixed this a very long time ago.  But the fix has never worked its way back into the official Arduino sources.

Here's the code with the fix, if anyone's interested.