I do command line ARM stuff on Windows/Linux. I have been using Codesourcery Lite, now I am using the ARM GNU embedded distro. I have just started working on porting my code to Due/SAM3X8E.
(1) A compiler toolchain
(2) An upload program
(3) Header files describing the Arduino pinout, memory-mapped data structures, etc.
(4) A basic implementation of libc / libm / libstdc++
The ARM toolchain provides the standard libs as well. You can get header files from the standalone ASF distro, but as you say there is a whole load of other stuff in there.
newlib is a bit more tricky. The standard newlib is too big for smaller MCUs. Various people have rolled their own version with varying levels of features, including Atmel in the ASF, but there is no common/portable newlib light/embedded. It would be difficult to do since embedded environments vary so widely.