Matrix problem

No. For row scanning, the resistors should be on the columns. For column scanning, the resistors should be on the rows. This is one of the problems with your current circuit: you are column scanning but the resistors are on the columns.

Whenever LEDs are connected in parallel, each led should have it's own series resistor. When scanning the matrix column-by-column, you are lighting a column of up to 8 LEDs at a time, so for each led to have its own series resistor, those resistors must be on the rows. When scanning the matrix by row-by-row, a row of up to 16 LEDs could be lit in parallel, so the resistors must be on the columns.

In terms of brightness, it will make no difference whether you scan by row or by column. You hit the same limitation either way: the 35mA limit of a '595 pin. If scanning by column, that 35mA can be shared between 8 LEDs, so around 4mA each. But the duty cycle is 1/16, so the average led current is 0.25mA. if scanning by row, the 35mA is shared by up to 16 LEDs, so only around 2mA per led. But the duty cycle is 1/8, so the average current is also 0.25mA.