Wakeup from the deepest sleep modes requires that the crystal oscillator restart, which takes about 65ms.