ground of Arduino and ground of power source

unlikely from what you say to be a capacitor problem, that is noramly an oscilation problem.

( A LDO is basically a high gain amplifier, so it oscillates great if you get the wrong output / input conditions )

sounds more like a blown uno or power supply.