Gotta be a better way...

lastchancename:
Rather than bring every wire back to the 7219's - why not wire the busses locally...
All the a-segment together, the b-segment etc all the way through to all the dp’s.

Then you'd be looking at 2x 7219 = 2x 16 wires from the display controller to the LED panel. 8x segments and 8x commons.
As noted above in #2 (and allowing for 16x 8-segment displays incl dp).

(EDIT - Tom George beat me to it!)

This would probably work best for me. I didn't even think about this solution haha. Thanks.

For everyone that asked, here are the links to the datasheets of the LEDs:

2 digit 7 segment

single digit 7 segment