Maybe it's designed to be used like that, although it still seems wrong to me. Why is the XBee connected to the serial port? Is it intended to send to the PC, or the Arduino, or what? The serial connection only has two end points, and if you have two devices trying to drive one end point then they're going to be fighting against each other. Anyway, you say that doesn't change the symptoms.
It's not uncommon with async serial comms for the receiver to need to receive a few characters to get synced to the bitstream. Perhaps this is what you're seeing? (Maybe aggravated by a serial line that has a distorted signal due to having two devices trying to drive it - but that's pure speculation.)