Ok, I looked at charlieplxing, but I dont see how that will help me, I am drawing 0.32Amps for each LED Matrix,
The point about a matrix is that you don't have all the LEDs on at the same time, you multiplex them so fast they look like they are on at the same time.
Otherwise you need drivers to provide the current both sourcing and sinking. This will allow you to control a whole row at a time.
You can not have full none multiplexed controll of each LED in a matrix. Tha act of wiring them up in a matrix means you have to multiplex them.