Is it possible to connect the output of the ntx2 to a high-power transistor to increase the signal?
That all depends on what "the ntx2" is, wouldn't you think?
And is there a library that allows me to use receiving data with my own program?
Data from what? Sent where? In an Arduino program or a PC program?
I will be sending serial data from the ntx2 transmitter and receive with a radio connected into my computer.
The HardwareSerial class (Serial is an instance) has everything you need to send data to the serial port. What to use on the PC is up to you. What languages are you familiar with? C# makes it really easy to read from and write to the serial port.