Frankly, I do not understand why the delay() and millis() shall be dependant on a crystal frequency entered in boards.txt.
The code for millis has to be told (at compile) the speed of processor clock. There is no way for that to be determined later (at run).
My understanding is for the clock frequencies from 32KhZ to 20MHz one shall be able to compile a delay() and millis() where one millis() unit is very close to 1ms
The Arduino folks wanted an efficient millisecond clock and as many PWM pins as possible. Those three goals drive the structure of the code and limit the clock speed choices.
The simple fact is that the code in wiring.c requires the processor clock to be a power of 2 (1 MHz, 2 MHz, ... 8 MHz, 16 MHz). To support arbitrary clock speeds requires code that is more complicated (less efficient) and, in some cases, giving up PWM on timer 0.