Not at all recommended for a beginner, but it is possible using only a capacitor and a resistor to get nicer sounding tones than a crude square wave. (even short sound samples
I used this technique for my 8 bit MIDI synth, made from Arduino.
Here's a bit of a song made with the synth:http://soundcloud.com/beefinator/diy-2-fixed
Here's a topic about this project (still a work-in-progress):http://arduino.cc/forum/index.php/topic,66726.0.html
Just as an example of what can be done with arduino and little other hardware.