So 255 at offset 255. Interesting. And as I mentioned, that does change it from 1.0 to -Inf, since the most significant byte (with the sign bit) is last.
How about larger payloads? 512, 1024, etc. And in the other direction, what if you read 128 or 64 at a time? (Depending on the available functions, may have to read it into a generic buffer and then memcpy it into the right place.) A byte at a time is of course simplest, and maybe it's not too slow.
You have to decide whether you want to try and fix the problem, find a clean solution, or just do a workaround.