Don't feel bad, I just bought a Leonardo today, after installing the innumerable drivers, tried to upload "Blink", but it failed at the last stage saying couldn't find the serial port.
I plugged my Lenny in, and Win7/64bit did what it was supposed to, and installed the drivers. I was able to upload a sketch first try, reading from Serial1, with XBee attached, and writing to Serial.
What OS are you using?