Maybe a simpler/different idea using below sudo code.
When the button is initially pressed make a copy of the time and set a inc/dec amount to 1/-1 depending on if your counting up/down
do
add the inc/dec to the time.
If the altered time value is <> 10 from the initial copy you made then set inc/dec amount to 10/-10
Check if button is still pressed and exit loop if not
loop