Ya, I have the same problem. I tried burning the bootloader to a 168 with the microcontroller option set to 8 and it just returned a bunch of errors (no surprise). Do I need a new hex file or something?
I am an absolute newbie, and havent successfully flashed a microcontroller yet (which i am desperately tryng to! (atmel168))
But heres the little i know with regards to your question -
Yes, you DO need a different .hex file.
Secondly, i dont think you can just use the arduino IDE to flash a 168 on the 8 setting because of "fuses". (Fuses are the little settings on the chip "the closest thing a microcontroller has to a config file") eg. They enable you to select whether the chip uses an external oscillator or the internal oscillator. And my clueless guess is that they are at different memory addresses on different chips (atmega8 vs 168).
Hence..you cannot fool the IDE to put a bootloader onto the 168 instead of the 8 just by replacing the .hex file.
Heres a really useful site which has helped me understand this a lot better - http://wolfpaulus.com/journal/embedded/arduino2.html
- it also has a link to the .hex file for the 168.