Can use HC595 to source current to anodes, just limit it to 8mA.
(Vs - Vf - Vc)/.008 = resistor value
Vs = 4.8V from HC595 - IOH drops with higher current - 0.2V loss at 6mA, could be as high as 0.52V (assuming similar drops with Vcc at 5V vs 4.5V in datasheet).
Vf = drop across LEDs (varies by color, blue typically ~3.7V)
Vc = drop across cathode sink device:
N-channel MOSFET will be lowest, NPN will be higher, ULN2803 even higher, like 1.3-1.6V.
ULN2803 only good for 500mA/IO, will be on the edge with 8mA/column, may need to use 2 IOpins/layer. Blue may not even turn on if Vf is 3.7V.
3.7V + 1.3V = 5V, greater than 4.8 that will be available.
Better off with MOSFET, with low Rds of 0.025ohm will have just 12.5mV drop & hardly any power dissipation:
P=I*2 * R = 1.28*1.28*.025 = 41mW.
ULN2803 only sinks current.
You could use them as anode "anti-drives" (just made that up):
+5 to current limit resistor to the anode.
When UNL2803 output is off, current flows into LED.
When ULN2803 output is on, current flows thru the ULN2803 instead, and LED stays off.
Then on the cathode, 4 ULN2803 IO pins/layer can sink the1.28A coming down the columns. Will be quite warm, maybe use a while chip per layer. I'd wire up just 1 layer to start, do some testing.
Still the question if there's enough voltage left for the LEDs to turn on.
I want to use nine 74HC595s, with 8 using your idea of the "anti-drives" for the anodes, and the last register using transistors or MOSFETs. I think I should use transistors or MOSFETs for the layers (cathodes) since the current is higher than the UNL2803 can handle.
What do suggest I use for the layers? Can you also explain your "anti-drive" setup in more detail? I know the outputs from the 74HC595 go the the UNL2803 then after that I am confused.