I got bored and just did the same thing with only 4 '595s (well, mine doesn't tell the time yet, it just counts up from zero one second at a time).
It has 3 88 matrices. My matrices have anode rows and cathode columns. With all respective rows connected, and columns controlled individually, I managed to make essentially an 824 matrix, which obviously only needs 32 pins = 4 74HC595s!
[edit]
(8 red + 8 green)
Just read that. I forgot yours was two-color. Mine is obviously only red.[/edit]