You basically keep the software going at all times. Connect parts one by one (the speaker first) until you hear the noise. I assume at least it's the speaker where it comes from.
To see if it's a combination of things: the moment you start hearing the noise, the component you added last is the suspect. Now start removing everything else one by one. If it's a combination, it should sooner or later stop. If not a combination, it persists until you remove that one offending part.
Those LEDs, are they driven by the pins or do you have a separate driver?