That's likely the problem. Does it sound like 50Hz? Or 100Hz, which more-normal for full-wave rectification?
If it's a high-pitch "whine" that's switching noise from a computer or switching power supply.
That may help. Try at least 1000uF. Or just get a better supply. Most "modern" power supplies are regulated and the regulator normally makes a very-good additional filter. Or, most switching supplies operate above the audio range which makes the noise easier to filter-out and it's inaudible anyway.
If it's a "charger" rather than a power supply there may be very little attempt to filter it because batteries don't care...
USB power from computers is often notoriously noisy, depending on the computer. It's not problem for digital but it can get into analog audio circuits.
Audio circuits can pick-up AC hum for the wires all around you, but then it wouldn't go-away with a battery.
If it's ground-loop hum, that's more difficult to deal with. That's unlikely since it's wireless and there's only one ground.
P.S.
Audio can be difficult. We can hear very-quiet sounds and we usually want a signal-to-noise ratio better than of 60dB (1000:1). ...We can't actually hear noise 60dB below the signal, but audio contains quiet parts and silent parts. So it's not REALLY the ratio...
Headphones, with the "speaker" right up to your ear, make noise easier to hear. If the headphones are "sealed" and blocking-out/reducing room noise, the electrical noise is ever easier to hear.