Need help: How to make my own 24x16 LED Matrix display

why not try to push the MCU to it's limits

Beside the display, you may need your MCU to perform other operations, like reading buttons, getting time from RTC etc. If the MCU is busy most of the time with the display, it may look unresponsive to the user.