I was using a 8ohm 0.5W speaker.One wire to ground and one PB3
And that's far too low of an impedance for an ATTiny's output pin. The speaker is trying to draw several hundred milliAmps from a pin that's not designed to supply that much current.
You need a resistor in series with with the speaker. I'd recommend around a 1K.
If you're lucky, you haven't dameged the chip.