Generally I don't like Frizing diagrams but in this case I was hoping the one you did would be helpful. You have confirmed what I thought: the diagram does not represent what is on the breadboard. You say the output is from the speaker, which is on the right of the Frizing diagram, but on the left of the breadboard in the photo. This makes the Fritzing diagram useless. The Fritzing diagram should be an accurate layout diagram showing actual connections. Not having the exact right component is a small detail that can be explained, having a completely different layout defeats the whole purpose of the diagram. As the pot and inductor represent components in the guitar, not on the breadboard, it would be clearer to show them off the breadboard, perhaps on a different breadboard, along with an explanation.
However...
By passive, I just mean that there are no powered electronics in the guitar.
I suspected that was the case, but as I have never had anything to do with guitars, electric or otherwise, I thought it best not to assume.
I don't understand why you say there there will be current flowing in the breadboard contacts. For this experiment, they are only tied to the electronics through their grounds. The signal wires are tied together but isolated from the MCU.
In the photo the problem is not there. In the Fritzing diagram in reply #9 you have the left hand contact of the pot going to 0V at the top of the breadboard and the inductor (pickup) going to 0V at the bottom of the breadboard. You might think those 2 points are electrically the same but they are not, especially from a noise point of view. They are a long way apart and in the path between them there are other circuits, especially the display, that draw current through the 0V connection. If you don't already know Ohms law please stop and learn it. The long path between those 2 connection points has resistance. Current through resistance results in voltage across the resistance, varying current through resistance results in varying voltage. Varying voltage equals noise. The noise will show up between the 2 connection points and find its way into the audio path.
What other processor points should have decoupling caps?
There should be a ceramic 0μ1 capacitor physically close to the power pins of the processor and the same for the display. The one you have is pretty close, but could be closer if you put it right next to the 2 power pins on the breadboard.
Yes, connecting the metal plate to ground quiets the chirp. Why is that?
You say the problems in the second circuit may be due to capacitive pick up. Can you explain what that means and how I can try to eliminate that problem?
Do you know how a capacitor works? 2 metal plates with an insulator between them? That's what you have in the second circuit, with the metal plate acting as one side of a capacitor and the rest of the circuit as the other side. 2 (or more) capacitors actually, with the plate as common between them. The plate picks up noise from the circuit and transfers it to the audio circuit. If you connect the plate to 0V you short out the interference signal and stop it getting into the audio circuit.
Unfortunately, I am at my limit of what to tell you. I still don't understand why the audio is even on the breadboard at all as nothing happens to it there. You say that a guitar with a built in amplifier does not have the problem; as I mentioned I have no experience with guitars but I suspect that the signal from a guitar pickup is at a very low level, making it susceptible to noise. If it is amplified at the guitar it will be both a much higher level and lower impedance, both of which will vastly reduce its susceptibility to noise. The answer might be a separate amplifier, in the same box of electronics as the processor and display, to amplify the guitar signal. If you do this then I suggest it should be in a grounded (connected to 0V) metal box with only 3 connections: input, output and power. The power should come from a separate power supply, preferably from a linear regulator, not a switch mode power supply (less noise), or batteries (less noise). Unless someone else can offer better help that's about all I have for you. I know there are people here who will have better answers (and might teach me something), obviously it's up to them if they wish to contribute.
Play around and see if you make any progress, but please, if you post a Fritzing diagram make sure it accurately reflects what you have on your breadboard.