You should be able to do it using only three pins, plus power/ground with the right driver chip! So no need for I/O expanders unless you really want them.
An example I built using the MAX7219 driver. Only works with common cathode displays though, so if you have common anode you'd need something else... but it was pretty easy to get running and I'm hardly an expert user!