Your circuit is missing the USB level conversion needed for the D- and D+ lines. You need to add 3.6V zener diodes for any USB interfacing directly to Atmel AVR chips. The 5V pin levels of the 32U4 is over driving the USB data lines. All the schematic references for Leonardo and Micro use the zener diodes for level conversion.
See this reference:
