Piezo is making a soft sound after having played a tone() - how can I stop it?

After noTone(), try setting the pin to INPUT and see what happens.

Yes, or write a logic LOW to the pin after the noTone.