Arduino UNO @ 20MHz

Today I tried getting an ATmega328P-PU to work at 20MHz. I downloaded the optiboot.zip file and copied the contents to my optiboot folder (I'm using Arduino-1.0.3 on linux). I made the necessary changes to boards.txt. I then started the Arduino ide and selected the "GVI High Speed (5V-20MHz-250k Baud)" board. Burned the bootloader with an USBasp, uploaded without errors. I then hooked up my ftdi usb/serial converter to the board and tried uploading the blink sketch. But no luck, it can't get serial communication. I tried another ftdi usb/serial converter but I got the same error:
avrdude: ser_recv(): programmer is not responding
avrdude: stk500_recv(): programmer is not responding

Just to be sure I plugged the chip in another board with a 16MHz crystal. Bootloaded the Uno bootloader and uploaded a couple of sketches, no problems there, so the chip is fine.

When I try uploading a sketch the led on pin 13 blinks a couple of times. So can I assume that the bootloader is working?