Help with increasing 8x8 LED Matrix brightness

The most current management I'm doing is switching it with the MOSFET which can take up to 9A

This means you will end up frying your shift registers.

I am having a bit of trouble down loading the data sheet for that FET, but almost certainly it is not a logic level FET and requires 10V to fully turn it on. You are supplying it with 5V and so it has a high, for a FET resistance. This is fine for small currents, but when a large current flows, with all your LEDs on the resistance drops a larger voltage and restrict what is flowing.

EDIT - just checked the data sheet and the FET is your problem, the Gate Threshold is only when the FET starts to turn on, not when it is fully on, you need 10V on the gate to get an on resistance of 0.4R which tells me it is a very old device newer devices have a much lower Ron ( resistance when on ) than this. So either boost the gate voltage or get another FET.