FIrst, what kind of XBees are you using? How are they configured?
How are you currently sending the data?
If I remember correctly, part of what the XBee library sends in each packet IS the ID of the sender and the ID of the recipient. Sending data from the master to a specific XBee will require reconfiguring the XBee for each send, and then configuring it back after the send. This takes time. During that time, data sent could be ignored by the master.
I guess each packet could contain an ID as to which player it's addressed to, and all the other players would just discard, if it's not theirs.
That's the solution I'd go for. Simple, easy to implement and debug.