The Arduino Nano doesn't normally use Optiboot. If you select the board "Arduino Nano w/ ATmega328" it will be expect the bootloader "ATmegaBOOT_168_atmega328.hex" at 57600 baud (see boards.txt). Optiboot runs at 115200 baud so it is not a drop-in replacement for the expected bootloader.
To burn a fresh bootloader on your Nano you should:
- Select Tools->Board->Arduino Nano w/ ATmega328
- Select Tools->Programmer->USBasp
- Tools->Burn Bootloader
If you want an Optiboot bootloader on your Nano you can put the following in a Arduino/hardware/boards.txt:
##############################################################
nano328O.name=Arduino Nano w/ ATmega328 & Optiboot
nano328O.upload.protocol=arduino
nano328O.upload.maximum_size=32256
nano328O.upload.speed=115200
nano328O.bootloader.low_fuses=0xFF
nano328O.bootloader.high_fuses=0xDE
nano328O.bootloader.extended_fuses=0x05
nano328O.bootloader.path=optiboot
nano328O.bootloader.file=optiboot_atmega328.hex
nano328O.bootloader.unlock_bits=0x3F
nano328O.bootloader.lock_bits=0x0F
nano328O.build.mcu=atmega328p
nano328O.build.f_cpu=16000000L
nano328O.build.core=arduino
nano328O.build.variant=eightanaloginputs
##############################################################
After that (and restarting the IDE) you can use the "Arduino Nano w/ ATmega328 & Optiboot" board type to burn a fresh bootloader and do your uploads.