Modifying HardwareSerial ... slightly?

Or am I looking at the wrong buffer

Yes.

That only tells you when the transmitt buffer is empty, not when the UART is empty. You will have to wait for another character time once that buffer flag is set before you can switch your TX / RX.