I am still not clear though what happens to the crystal / frequency as you reduce the voltage
Nothing happens to the frequency. What will happen is that at some pointing will stop, or fail to start when that voltage is applied, or will not be able to work at as higher temperature. Or more worryingly some of the latches gates and flip flops inside the processor might skip a clock cycle and the processor starts to screwup. You might not notice this at first because you might not be using that Instruction. But it will start to stutter.
All in all not a place you want to be.