Simple Volume Control

I am surprised that you have not got AC coupling. Are you sure that your audio signal is confined in the 0 to 5V range? If not it will distort, unless you connect Vss to a -5V supply.