You need to have some mechanism to synchronise your receiver with the transmitter.
Imagine your tranmitter sends out a continuous stream of groups of three bytes:
RGBRGBRGBRGBRGB
^
Now, your receiver expects to get R followed by G followed by G.
But imagine it doesn't start running until the point indicated by the arrow.
It receives the value meant for the green channel, and puts it into the value for the red, then it gets the value meant for the blue channel and puts it into the green, and so on.
You could have a missed character, or a corrupted one.
There are lots of ways around this, but they need some thought.