I guess if you switch to attiny84, you might be able to use the internal comparator to wake the MCU from sleep. You would need to use some high-pass filter on the pot output to allow the comparator to detect when a change in the pot setting occurred and exceeds some threshold you could set with a trim-pot. But you would need to detect a change in either direction, and I'm not sure how that could be achieved.