1 pair each. Lot of current draw in each part, caps add local current buffering.
I'm with Crossroads in both caps per chip but minimum should be a 0.1uF per chip and maybe 10uF a pair.Are you intending on driving both colours from same chip of different chip. Same seems possible but from different will probably be a headache.
I'm also curious about driving dual color matrix. Typically a matrix would have a shared cathode and 2 sets of anodes for the 2 colors, so 24 pins total on the matrix.