Led panel

Where are the resistors! You need about 270R in line with each column.
You never said this was a charleyplexing circuit, you drive those very differently from a normal matrix.