It could be the wrong ESR rating for the actual regulator used. My Leonardo has a 1117 5V
regulator and I haven't noticed problems (but then I wasn't looking for them or using battery
power).
Low dropout regulators usually have tight requirements on their output capacitors,
a minimum and a maximum ESR value for instance, to ensure stability across the full
voltage and current range.
If these are knock-off boards the wrong regulator or capacitor may have been
substituted.
Ceramic caps use the same materials as piezo transducers so will sing like this. You don't
use them in audio pre-amps because they act as microphones.