2-way Serial Communication between Arduinos

I'm attempting to set up two arduinos to send data back and forth between eachother. I can successfully send data from #1 to #2 but as soon as I connect a second wire from the TX of #2 to the RX of #1, #2 appears to stop receiving data completely. I've looked into it a little further and I've found that #2's TX pin is held at a digital HIGH. If I connect this pin to another pin that is HIGH everything works fine, but as soon as I connect it to a LOW pin everything shuts down. Can anyone help me to understand how I can set up 2-way communication between the two arduinos?

Thankyou

Rx to Tx, Tx to Rx, Gnd to Gnd. Pretty basic.

Oops. I forgot the GND to GND part. Thanks.