The Arduino core uses Timer 0 to keep track of time. You will get an interrupt about once every millisecond.
Perhaps you can disable the Timer 0 interrupt before going to sleep and re-enable it when you wake up. Of course that means the millis() timer will stop running while you are asleep.
Another option is to use a deeper sleep like Power-save Mode and use Timer 2 to wake up. If I'm reading the datasheet correctly, Timer 2 is the only timer running in Power-save Mode.