Thats because your compiling against the Arduino libraries. They are always included.
If you compile it separately (with a Makefile) then there is no overhead.
Is that a difference in "cc -o whatever pins_arduino.o HardwareSerial.o ..." vs "cc -o whatever -larduino" (where libarduino.a is the output of "ar <er, something...> pins_arduino.o HardwareSerial.o ...")?