Here’s my experience using avr-gcc version 4.3.4, built from source, on my Centos 5.5 Linux workstation. See Footnote.
When I executed “make atmega328” using arduino-0022/arduino/bootloaders/Makefile, the size of the object file was too large (532 bytes)
To get it below the 512 byte limit I did the following:
I changed line 61 of Makefile from the following
override LDFLAGS = -Wl,$(LDSECTION) -Wl,--relax -nostartfiles
to the following
override LDFLAGS = -Wl,$(LDSECTION) -Wl,--relax -nostartfiles -nostdlib
Now, with my compiler, that gets the object size down to 490 bytes.
Before going any further, there is a bug in that version of optiboot.c that has been documented elsewhere. I really, really (really) recommend that you do the following, as I did:
I changed line 216 of optiboot.c from the following
// asm volatile ("clr __zero_reg__");
to the following
asm volatile ("clr __zero_reg__");
This must be the first executable statement in main().
This statement requires an avr header that was not previously needed, so I inserted the following after line 71 of optiboot.c
For me, that brought the object size to 492 bytes.
The code is very tightly written, and you won’t have oodles and oodles of room to add any enhancements, but maybe this will give you a chance to start.
There have been other changes in optiboot since the last Arduino release. If you really want to work with the latest, go to the Optiboot Home Page and follow the instructions to check out the latest optiboot source. That code leaves less headroom than the code in arduino-0022, so you may have problems shoe-horning other stuff into it.
Bottom line: If you have a different version of avr-gcc, you may not see the exact same results. Do I have to say it? IWFMYMMV. (It works ror me, your mileage may vary.)
Footnote: Actually, I created a new working subdirectory under the optiboot directory and copied everything there. That way, I could experiment to my little heart’s content in the working directory and still have the originals with which to compare.