You need to control PNP transistors (or logic level, low Rds P-channel MOSFETS) to drive current into each anode.
Then buffer the individual cathodes with ULN2803, or TPIC6B595 shift register, so that full 20mA can be sunk thru each segment.
Your coding is also important - need to maximize the on-time for each digit.