Yes, that's it. If you remove the 100K pullup, and keep the 100K pulldown - to keep
the transistor turned off, and then capacitively couple the input signal, as others have
mentioned, then the transistor will stay off when no signal is present. Then, you also
don't need to capacitor-couple the speaker.
However, you'll only ever get square-wave outputs [ie, rasty sounding things] from
your speaker.
It is possible to get sweeter sounding audio using the Arduino. Would require going to
a true Class A amplifier ckt [or more complex Class B, etc, as Krupski mentioned], and
then using high-frequency PWM, modulated at a lower audio rate, and then using
low-pass filters ahead of the amplifier to smooth out and anti-alias the PWM.