I'm reproducing your results, so I'm now working out why. I've added some debugging displays, which are interesting:
Awake!
Got key: 1
Awake!
Got key: 4
Awake!
Awake!
Got key: 4
Awake!
Awake!
Awake!
Got key: 4
First, we get an "awake" message (at the end of the sleep routine) which shows it wakes up but doesn't detect a valid press (for some reason).
Second, I can press key 4 and it doesn't always wake up. So, it seems there are two problems. 