Here's my vision: 15 shift registers, each sinking current from a 4-digit display, daisychained with SPI.transfer to load them with data.
4 PNP or P-channel MOSFETs, one at a time turned on, to allow one digit from each group to be on.
charlieplexing -thought I saw that mentioned...
