Why not just use the readily available cascadeable modules using MAX7219s?Put two together and blank out (simply do not activate) the digits you do not want.
Try to search on digikey.
Are the displays common anode or common cathode ?Some driver chips can sink only, eg pcf8574, so are suitable only for a common anode device.
Ht16k33! This chip can drive your displays and your individual LEDs, and also scan your pushbuttons.