PeterH:
PWM ... can be ended by digitalWrite(). It would also be ended by analogWrite(0) or analogWrite(255).
Indeed. I can't vouch for digitalWrite(), having never tried it. But analogWrite'ing 0 or 255 would do it. In fact, it would be interesting to see if things change between analogWrite(0) and analogWrite(255) - one drives the output close to ground, and the other near Vcc - I'd want to bet that analogWrite(0) would yield a quieter output. Or, you could just comment out all the analogWrite()'s altogether.
Remember, you've got a whole lot of digital activity going on in the Arduino and in the PC, and even in the sound card. All of that activity influences Vcc, however slightly, and that influence won't typically be harmonic with your output signal, making it easy to hear. My favorite theory right now is that you're hearing audible components of power supply noise, influenced by all that digital activity.
As to whether the breadboard could be a problem: It's not much of a noise source in itself, though it might pick up some ambient electromagnetic fields. I think it's more likely that using the breadboard means that you use long, loose wires to connect the components. I've seen the performance of a number of analog projects improve when I replaced pre-cut jumpers with cut-to-length telephone wire. Loops of wire act like antennas, and, probably worse, have a higher impedance than short wires or printed-circuit traces, leading to small but noticeable voltage drops across them.
You might get improvement by using single-point grounding. Bring a ground from the Arduino to a convenient spot on the breadboard, and then run every other ground to that same point. We sometimes want to think of every ground as being exactly the same; however, when we run a ground from one device to another in daisy-chain fashion, we accumulate current - and hence unwanted voltages - in the chain of conductors. I've seen analog circuits improve, sometimes considerably, when I reminded myself to do this.
Finally, I encourage you to use power-supply decoupling capacitors very close to any analog components that draw power from the supply, like the amplifier IC, if you don't have them already. I'd recommend an electrolytic and a ceramic capacitor in parallel - maybe a 10 uF and a 100 nF. Texas Instruments' datasheet for the LM386 quotes a power supply rejection ratio of 50dB, but mentions that the test is performed with a 10 uF bypass capacitor. I can't quite do the math just now to figure out whether 50dB of rejection is really good enough for this application. Bypass capacitors might not help much, or they might make a huge difference, depending on the actual source of your unwanted signal.
The reality is that you're trying to get very good analog performance from a digital circuit. That's not typically as easy as we'd hope - you have some work ahead of you.
polish:
... I think I'm overthinking this ...
You're overthinking it? I'm defenitely overthinking it, and it's not even my project.
You haven't said whether the power to your analog section is the same supply that runs the Arduino. Is it?