SOLVED:Arduino Due Delay Effect - Noisy playback *only* with variable delay time

Ok, this is not due to EMI. The only explanation I can think of is that noise comes from the map() usage:

map(analogRead(A2),0,4095,1,20000))

If analogRead(A2) = 0, the result should be 0 (0 is the minimum buffer Index)
If analogRead(A0) = 4095, the result should be 19999 (19999 is the maximum buffer Index)

You should Serial.print the result of the map() function in your code to see where is the issue while you turn the pot. BTW, select Serial.begin(250000).