I couldn't figure out how to get rid of the noise, so I found a product from RuggedCircuits called RuggedAudio that I use. http://www.ruggedcircuits.com/html/rugged_audio_shield.html
This shield wasn't specifically designed for the Due, so to use it I
* cut traces 13, 12, 11
* solder in the SPI header on the card
* remove the card to flash the Due
I use a SanDisk SD card.
I scale the audio to 12 bithttp://forum.audacityteam.org/viewtopic.php?f=26&t=36498#p96024
(instructions to do this in audacity are in one of the posts)
Because of the 12 bit audio there is some static noise, but this solution has been acceptable for me.