Serial.print() converts a value to ASCII text (human-readable). Depending on the type of data we give it (byte, int, long, etc.) and the actual value, it may produce one or more bytes, one byte (character) for each digit in the value.
Serial.read() reads a single byte.
Sounds like the two ends are operating inconsistently, yes? I'd look at that before I'd suspect the XBees.
As ever, post your code, some days we are not so good at debugging what we cannot see. 
As an example, if on the sending end I do
byte myValue = 42;
Serial.print(myValue, DEC);
That will convert the binary value of 42 (which in data memory is contained in a single byte) to ASCII characters, namely '4' and '2', each of which is a byte and which actually have values of 52 and 50 respectively (see your ASCII code chart).
Now if the receiving end does
byte myReceivedValue;
myReceivedValue = Serial.read();
I will end up with the value 52 in the myReceivedValue variable, not what I was perhaps expecting.
The more complete answer to the question is that it can be done either way, but the conversion (if any) on one end has to be undone at the other end. I can send binary values and read binary values, or I can send ASCII text, but then on the receiving end I need to convert a string of characters back to a binary value in memory.