You missed a line on the datasheet didn’t you?
Reading this bit field will return the value of PORTx.DIR.
So you set the TX bit to be output.
|= sets something to the BITWISE OR of current value and what you’re assigning to it. So you just set it to 0x02 via the assignment to the DIRSET register just before this.
Now it gets read in, and bitwise or with PORT_INT2_bm. 0x02 | 0x04 = 0x06
Then you assign that back to the DIRCLR register… setting both bits to inputs!
NEVER USE |= WITH THE ___SET/___CLR REGISTERS!
I see a lot of careless use of thse |= and &= operators in Arduino-land. They are a syntactic sugar for a read-modify-write operation (which is not interrupt safe) on all registers except the lowest 32 (the so-called “Low I/O space” - on modern AVRs like the tinyAVR 1-series, only the VPORT registers and 4 GPIO/GPR registers are in the low I/O space) - on those, and ONLY those, a |= with only a single 1, or a &= with only a single 0, where the register you are trying to flip a bit in AND the bit you are trying to flip both known at compile-time, will get compiled to sbi or cbi (set/clear bit index) which is an atomic instruction that is interrupt safe. (executes in 1 cycle on modern AVR, 2 on classic ones) - but if one of those isn’t known at compiletime, you’re back to a read-modify-write.
The___SET/CLR/TGL registers should be used with normal assignment only, eg:
PORTA.DIRCLR = 1<<2;
Or if you insist:
PORTA.DIRCLR = PORT_INT2_bm;
(as an aside, on classic AVR, PORTx |= 1<<x takes 2 cycles to execute, and 1 word of flash. PORTx |= (0x03); (or any other constant) is 3 cycles, 3 words of flash, and has 2 cycles during which an interrupt could change that value out from under you. That’s single cycle for read and write, because it’s located in the I/O space (the lowest 64 registers).
On modern AVR, basically nothing is in the high IO space besides a the protection register, maybe the stack pointer, status register… (on the plus side, the registers are logically laid out, rather than looking like they involved a monkey and a dart board in choosing register addresses) - and the low IO space is filled with VPORTs and the handful of GPRs. So a read-modify-write cycle is 6 clock cycles and 5 instruction words! (LDS is the only instruction that got the short end of the stick when they refreshed AVR instruction set - it went from 2 clocks to 3 - I think that was the price we paid for ST and PUSH getting made single-cycle)
Anyway, as I was saying, people throw around |= and &= - including in cores, where the author really ought to know better, especially when they’re initializing the hardware for first use, so they know the register was initially set to 0… and yet, I see lines where they set 2-3 registers at a time with |= (which is always a read-modify-write), when they knew damned well the register was previously set to 0 by power-on reset.
Why are you writing your own UART routine, btw? You know that megaTinyCore gives you a normal Serial object to talk to the UART right?