Looks like a really interesting project... Could you by any chance post the interrupt code? Always interesting to se how people solve the problem whit multiplexing lots of leds...
Oh and what method do you use to set the state of the pins, do you use digitalWrite or do you set the pins using direct access to the registers... The later one is much much faster and is good when you have to do it fast and often.
/Jon