Start with the Arduino IDE, go to other tools if it becomes limiting. Can't say that's happened to me yet, although I've used WinAVR and to a lesser extent, AS6. Programming can be done as close to the hardware as desired with any of those. Libraries can be used or not. Practically, I can't expect to reinvent all the wheels, although I certainly do enjoy reinventing a few for the reasons you mentioned. So I usually end up with a combination. Often, if I am adding some peripheral hardware, I will write my own library even though I'm sure I could find multiple examples out there.
A recent project used four of my own libraries, one from the Arduino playground, one of the libraries included with the Arduino IDE, two from elsewhere around the web, and two of the base AVR Libc libraries. Plus some of the standard Arduino stuff like Serial.