Transistors are needed so you can turn on a whole layer at one time, so 320mA of current needs to be controlled.
Without transistors, you are limited to 1 LED a time being controlled at a time (with a traditional 16 columns and 4 layers), and updates will occur a lot slower; 64 LED on/off cycles per cube compared to 4/cube.