I did the same with a MAX7221 for my fencing scoring machine.
I check every 1/4 second to make the colon flash on/off, and every 4 checks update the seconds and other digits as needed.
The rest of the time I am checking for button presses, RF commands, etc.
Easy to change that for 1/10 second checks, still plenty of time to read slow (by comparison) button presses.