I modified the ATmegaBOOT_168.c file found in the bootloader section to use 9600 baud and a different bootloader button (which isn't even used apparently). I built the new file using the Makefile as "make ng" and it created the hex file. When I tried to download the hex file to my MEGA168 (also using the Makefile) via my AVRISP 2, I got this error:
avrdude: ERROR: address 0x4010 out of range at line 129 of ATmegaBOOT_168_ng.hex
avrdude: write to file 'ATmegaBOOT_168_ng.hex' failed
I checked the hex file and sure enough, there are addresses outside of the $4000 (16K) range. How did that happen? Have others seen this and how can I fix it? The pre-existing hex files in that bootloader directory did not have this problem, they fit in fine.
Let me append to this question. I am using avr-gcc 4.3.0 (latest MacPack distro). It could be that this version of the compiler is more verbose than expected. Could I yank the debug stuff out safely?
And now I solve the whole problem. I'll explain in case others have this problem.
On the Mac you can put AVRMacPack which handles tons of configuration for you. It installs both a gcc 3.X and 4.X version. I have been using the 4.3.0 compiler. I used the following MacPack command to change to the 3.4.6 gcc:
"avr-gcc-select 3"
I then recompiled and everything fits and programs. The bootloader appears to be working, now to figure out how to put code into the device!