At a certain moment you can only use whole dividers of the clock speed as only then you can get the timing right over the 10 bits of a byte.
Note that if there is a inaccuracy of 10% you will fail at least one bit (draw a timing/sample diagram) and at higher speeds every cpu ccyle becomes a larger percentage..