Probably a Carriage Return (CR, '\r') and Line Feed (LF, '\n') are transmitted if you press enter.
You can write the code to the serial monitor in hexadecimal notation, but you have to do that before the '0' is subtracted.
when you say "You can write the code to the serial monitor in hexadecimal notation, but you have to do that before the '0' is subtracted." you mean in the serial monitor i should enter the value in hexadecimal, without the subtraction of zero, right?
if so, i tried tat too. but then all the outputs in the serial monitor are "INVALID!".
Lets start with the simple stuff, first. In the Serial Monitor, what line ending do you have selected? I'm guessing that it is not none, so you are sending something like "0" and the cr and lf are what is invalid.