pascalm:
For a project I have for generating audio signals, this noise was unacceptable, so I cut out the regulator and powered everything with an external LM7805.
Crude solution.
As fungus has pointed out, it's C12, a 10uF ceramic cap.
It is there to stop the 5volt regulator from oscillating, but apparently it's not very good at it.
I would just piggyback a second smd cap on top of C12.
If you're not good at smd soldering, try a 47uF electrolytic cap in the 5volt and ground headers.
If it works, you could solder that to the bottom of the board
Leo..