If I am reading the top schematic right you are using a 74HC595 to select your anode (rows) on the matrices so the 595 is the source of current. I thought they were terrible current sources. Won't you need a PNP transistor or a p-channel MOSFET in there somewhere as a better current source? And why not just us a 74HC138 decoder instead of a shift register anyway - that provides high on all lines except low on the selected line which is a natural fit for the current source.