Any ideas on what I am doing wrong?
Not without you posting a good picture of your wiring.
On the packaging of the diode I am using it sais: 1N4001
That is a rectifier diode and while it will work the response is not so fast, so try a small signal diode like a 1N4148.
Also with a diode your audio signal needs to be above 0.7V for the diode to conduct.