TXEN in HardwareSerial.cpp - Where is it defined?

It's an AVR define.

From the ATMEL ATmega (basically mega48 thru 328) doc8271.pdf:

20.6 Data Transmission – The USART Transmitter
The USART Transmitter is enabled by setting the Transmit Enable (TXEN) bit in the UCSRnB
Register.

complete notes and usage are in the (20 meg pdf, over 300 pages) doc.