The minimum resistance for the Arduino is 125 Ohms. The "absolute maximum" current is 40mA. From Ohm's Law, 5V/0.040A = 125 Ohms. 180 or 240 Ohms is more reasonable because you aren't pushing the Arduino to it's limits.
You can put a 120 Ohm or greater resistor in series with the speaker, but it won't be very loud because most of the power will go to the resistor, and there is not much power with 5V into 125 Ohms to begin with.
The easiest thing to do is to use a regular "powered" computer speaker.
Or, you can use a transistor, or MOSFET to boost the current (and to use with a higher voltage, if you wish). With a transistor or MOSFET, you don't need a current-limiting resistor. But, you might want to add a capacitor to block the DC from the speaker.
With 5V and 4 Ohms you can't get 5W... The maximum you can get without using a higher voltage is about 1.5W, and that's with a full-power square wave, not voice or music. (You can get about 3W with 5VDC into 4 Ohms, but DC doesn't make any sound.