480ma@5v is the max per column, and not really PWM, its the same as all LEDs on, 0ma with all LEDs off, and somewhere in between for PWM.
I have a test rig that uses 8 RGB LEDs, and I've hooked it up to an external power supply, because that sounds like too much current for the arduino.
Your diagram doesnt include any resistors, or IC's or other helpful info, but from what I can tell, controlling RGB LEDs with PWM is going to be limited with a common cathode setup, common anode setup doesnt seem to be as limiting in driving options.