Custom Leonardo board no USB?

Hi, I have made my own version of the Leonardo, and I am having USB issues. Plugging it in to a windows computer - nothing happens! If I hold down reset, I get the "device not recognized error", but if I don't hold down reset, Windows acts like nothing is there.

Schematic:

Fuses
Extended: FB (If I program it as CB, then press read again, it comes out as FB?)
High: D8
Low: FF

Using Caterina-Leonardo.hex, uploaded with AVR ISP MKII

The board itself is working, if I compile a blink sketch and flash it directly it works great. It's just that when I plug it into a computer, nothing happens.

Any ideas?