It is worth trying. Two things would be needed: core files and a bootloader.
Based on the ideas described here: Alternate CORE files for Arduino I would expect that the core files for the sanguino or the 1284 would work as is because the chips are pin compatible.
I checked out the optiboot sources and made a rough attempt to build a booter for the atmega168p. This patch file describes my modifications (patch -p1 < patch_file when positioned in the optiboot root dir applies them):
diff -r cc5e4843feec optiboot/bootloaders/optiboot/Makefile
--- a/optiboot/bootloaders/optiboot/Makefile Tue Apr 03 00:51:23 2012 -0700
+++ b/optiboot/bootloaders/optiboot/Makefile Wed Oct 03 00:10:06 2012 +0200
@@ -265,6 +265,25 @@
atmega1284_isp: EFUSE = FD
atmega1284_isp: isp
+atmega164: TARGET = atmega164p
+atmega164: MCU_TARGET = atmega164p
+atmega164: CFLAGS += '-DLED_START_FLASHES=3' '-DBAUD_RATE=115200'
+atmega164: AVR_FREQ = 16000000L
+atmega164: LDSECTIONS = -Wl,--section-start=.text=0x3e00 -Wl,--section-start=.version=0x3ffe
+atmega164: $(PROGRAM)_atmega164p.hex
+atmega164: $(PROGRAM)_atmega164p.lst
+
+atmega164_isp: atmega164
+atmega164_isp: TARGET = atmega164p
+atmega164_isp: MCU_TARGET = atmega164p
+# 512 byte boot
+atmega164_isp: HFUSE = DC
+# Low power xtal (16MHz) 16KCK/14CK+65ms
+atmega164_isp: LFUSE = FF
+# 2.7V brownout
+atmega164_isp: EFUSE = FD
+atmega164_isp: isp
+
# Sanguino has a minimum boot size of 1024 bytes, so enable extra functions
#
sanguino: TARGET = atmega644p
diff -r cc5e4843feec optiboot/bootloaders/optiboot/makeall
--- a/optiboot/bootloaders/optiboot/makeall Tue Apr 03 00:51:23 2012 -0700
+++ b/optiboot/bootloaders/optiboot/makeall Wed Oct 03 00:10:06 2012 +0200
@@ -18,3 +18,4 @@
make mega
make atmega88
make luminet
+make atmega164
diff -r cc5e4843feec optiboot/bootloaders/optiboot/optiboot.c
--- a/optiboot/bootloaders/optiboot/optiboot.c Tue Apr 03 00:51:23 2012 -0700
+++ b/optiboot/bootloaders/optiboot/optiboot.c Wed Oct 03 00:10:06 2012 +0200
@@ -303,6 +303,9 @@
#elif defined (__AVR_ATmega644P__)
#define RAMSTART (0x100)
#define NRWWSTART (0xE000)
+#elif defined (__AVR_ATmega164P__)
+#define RAMSTART (0x100)
+#define NRWWSTART (0x0)
// correct for a bug in avr-libc
#undef SIGNATURE_2
#define SIGNATURE_2 0x0A
diff -r cc5e4843feec optiboot/bootloaders/optiboot/pin_defs.h
--- a/optiboot/bootloaders/optiboot/pin_defs.h Tue Apr 03 00:51:23 2012 -0700
+++ b/optiboot/bootloaders/optiboot/pin_defs.h Wed Oct 03 00:10:06 2012 +0200
@@ -44,7 +44,7 @@
#endif
/* Sanguino support */
-#if defined(__AVR_ATmega644P__) || defined(__AVR_ATmega1284P__)
+#if defined(__AVR_ATmega164P__) || defined(__AVR_ATmega644P__) || defined(__AVR_ATmega1284P__)
/* Onboard LED is connected to pin PB0 on Sanguino */
#define LED_DDR DDRB
#define LED_PORT PORTB
This creates a new make target atmega164. The booter will be placed at the end of the 16K flash, leds are the same as on the 644...
This compiles a optiboot_atmega164p.hex. No idea what other changes are still needed. E.g. the atmega164p has a page size of 64 words while the atmega644p has a page size of 128 words. I don't know whether the optiboot code and/or the avrdude.conf file automatically deal with this. Unfortunately I don't have an atmega164p to try, so I am a bit blocked here.
Maybe somebody with some optiboot code knowledge can jump in here?