I would suggest PNPs for common anode layers, and open drain shift registers, TPIC6B595 type, on the cathode columns.
Can use 5 '595s, 25 for cathode, and 5 to pull the PNP gates low to turn them on.
Each TPIC6B595 can easily sink the 160mA from 8 LEDs all on at once at 20mA if want them good & bright.