Use cd74ac164 to source current to individual columns, and N-channel MOSFETs to sink current from common layer;
Or, use '595 to drive P-channel MOSFETs to source current to common layer and TPIC6B595 to sink current from individual columns. TPIC6B595 can only pull low, so can be used to pull gate of P-channel MOSFET low to turn them on.