NEVER short out capacitors like that - you may destroy the board. (LiPo batteries
will put out 50A or more compunction, enough to vaporize all the power traces).
The Leonardo doesn't have a switched-mode regulator according to the schematic,
so its a mystery. Decoupling caps should have a constant voltage across them so won't
make enough noise to be audible in my experience.
I would suspect the inductor L1 on the USB supply part of the circuit - except you are on
battery power...
You can investigate the precise origin using a pen or similar - place against suspect
component and other end to your ear (carefully!).
You could check any AC component on the supply rails with a multimeter on ac-volts
range - it should be very low (< 0.1Vrms)