Out of interest, what's the meaning of the data? Latitude and Longitude?
At any rate, your three bytes need to be processed piecewise.
1st byte has 2 parts, a sign and, from the looks of it, a single number from 0-9
Second byte, two BCD digits
Third byte, two BCD digits.
I'd start as follows:
1st Digit, AND with 0x0F, multiply by 100, store in result
2nd Digit, AND with 0xF0, shift right by 4 bits, multiply by 10, add to result
2nd Digit, AND with 0x0F, add to result
3rd Digit, AND with 0xF0, shift right by 4 bits, divide by 10, add to result
3rd Digit, AND with 0x0F, divide by 100, add to result
1st Digit, AND with 0x10, if not 0, multiply result by -1.
Be advised, there are probably faster ways, but this stepwise process is clear.
Go ahead and code this, come back with the result if it doesn't work.
// Put the number together as an integer.
unsigned long number;
number = msg[0] & 0x0F;
number = (number * 10) + (msg[1] >> 4);
number = (number * 10) + (msg[1] & 0x0F);
number = (number * 10) + (msg[2] >> 4);
number = (number * 10) + (msg[2] & 0x0F);
number = (number * 10) + (msg[3] >> 4);
number = (number * 10) + (msg[3] & 0x0F);
// Then divide by 100.0:
float value = number / 100.0;
// Then apply the sign:
if ((msg[0] >> 4) == 1)
value = -value;