I used an ATTiny85 to send serial data and an Arduino Uno to receive it. First I got some strange gibberish too until I pulled the ATTiny's VCC out by accident. Without VCC on the ATTiny it sent what I wanted, after reconnecting VCC it was gibberish again. There were no pins except Tx, Rx and GND connected.
EDIT: Don't do this, a 100nf capacitor between the signal line and GND should do the trick.