I need to know some encoding format to encode a serial stream. The idea is encode it in a hard way using maths to rebuild it at the receiver side.
If the RF link fails or the IR fails, or noise appears and currupt the packet, the receiver must have some way using maths to rebuild the corrupted stream. :o
I need some idea to encode this stream and to decode it at the receiver side. ;)
To accomplish the goals you stated, you need to do more than encode your data stream. You need a protocol that detects missing or erroneous data and can retransmit it if necessary.
You might want to do some research on reliable communication protocols.
Forget maths, just send the same data over and over and have the receiver see what the most common byte was sent at each position.
Or, take a college-level engineering course in communications theory. ;) Basically, what you're asking to do is not easy.