An easier approach to the whole thing would be to do the conversion to Hex-values by some simple bit-shifting.
Since encoder0Pos is an int we will have a maximum of 4 Hex-bytes
byte HEX0; //The lowest nibble (4bit = Hex-value ) of the encoderpos
byte HEX1;
byte HEX2;
byte HEX3; //The Highest nibble (4bit = Hex-value ) of the encoderpos
//We clear the upper 12 bits of the encoder pos. The lowest Hex-byte is just the bits 0-3 of encoder0Pos
HEX0 = (byte) (encoder0Pos & 0x000F);
//The higher the nibbles need to be shifted to the right after being masked
HEX1 = (byte) ((encoder0Pos & 0x00F0) >> 4);
HEX2 = (byte) ((encoder0Pos & 0x0F00) >> 8);
HEX3 = (byte) ((encoder0Pos & 0xF000) >> 12);
I guess you know that these bytes are not the Hex-Numbers
characters ?
They are just bytes in range 0..16. (In case you want to send the characters '0'..'F' over the serial line)
Eberhard