Sorry long post.
Whether it should be on or off by default, I can't say
Okay. Off 
- I can see an argument from both sides that is equally valid. ...
Sorry, I disagree. 'Magic' should be off by default.
Experienced users will be able to find it and switch it on, and inexperienced people won't have to worry about yet another thing. I don't want new users to spend time understanding features which aren't essential.
I do think it should be customizeable whether you want it (as well as to allow further extension for user-created or third-party libraries).
That is even more complexity, but okay with me, as long as it's off by default, and it is appropriately prioritised.
this (and other tools) could be implemented best as macro commands to external scripts
I'd like to understand what the need is, and who it's for before focusing on implementation.
Please write down the stories, or use cases, and actors which benefit by completion. Then identify the actors or use cases which are not addressed by Eclipse-for-Arduino.
Now who's overcomplicating things?
I have seen folks ask for use cases on this forum, so I assumed folks were okay with the terminology.
They are just techniques to capture needs, and the users who will use the solutions.
There is always more desire for new functionality than development resources. IMHO, it is helpful to be able to group and prioritise the desires.
Use cases or stories are popular, and effective, techniques for focusing on "needs in context", rather than lists of features. Use cases or stories should be recognisable by their users as things they want to do. Many other techniques fail to get the coherence of these approaches.
Actors is a developer short hand. It recognises that we are users, but sometimes we are doing such different things, that we need to separate the jobs. For example, I am 'acting' as a user of a computer on a network, but sometimes I am 'acting' as the network admin. My needs in these two situations are so different, that it is helpful to partition them, and treat them as different groupings of functionality.
I am always interested in better ways to be clear about needs. So I'm happy to look at alternatives.
You seem to be fairly concerned with newcomers and how they'll react to the IDE as such. I am not suggesting that a full blow-out of the IDE is needed or wanted, but a few changes that would greatly speed development are not out of order here.
I have no problems with improvements.
As long as 'Magic' is off by default, or so perfect it makes less sense to be off.
I would like the IDE to be stripped down and made simpler to use.
For example, if it could solve things which are error prone, like automatically recognise the port & Arduino board.
... in order to figure out a particular function or method call, I have to wade through a lot of documentation, some of it incomplete, most of it not even listing the argument types being passed, or the return types. For this, I have to then go to the header files. ...
I have no problem with a clean, elegant, mechanism to deliver faultless information, but maybe an initial step is capturing the information? For example, have all of the core libraries got good variable names for the function prototype information? If not, what's the process for improving it?
That's the problem - having this information at your fingertips, inline - personally I would think would be helpful to both experienced users and newcomers alike (provided that the newcomers understand what is being shown and why, and how to use the feature). There would have to be education on these concepts to new users; but that should be a part of the documentation for the Arduino IDE and the Arduino itself.
I think the need, for new users, is to avoid spending more time reading documentation, and getting educated until they feel the desire. The IDE is a means to an end, not an end in itself.
Let me say one more thing. I worked for a large-ish (1000+ staff) company which was growing at almost 50%/annum, probably comparable to the growth of interest in Arduino (I haven't seen the numbers). For that company, it meant the majority of staff had less than two years experience with company systems and processes, all the time. Sometimes we had to focus on simplicity, because the cost of supporting complexity grows too fast.
I accept that new users aren't the only users, but IMHO, a key attraction of Arduino is its simple, uncluttered IDE.
I have no issues with having extra facilities which experienced users can access. The children I work with will find extra facilities in a couple of hours if they are very useful (these things spread like wildfire if they are good).
I am pleading that the under-5-minutes-from-zero-to-blink is not made more complex, but easier.
If the current growth in Arduino users continues, more new users will experience the Arduino IDE for the first time in the future than have experienced it so far.
GB