it says that each output can source 5 to 100mA
I must admit it is well hidden. The clues are when it says:-
The STP08CP05 guarantees a 20V output driving
capability, allowing users to connect more LEDs in
series
Now you can't get 20V output from a chip you are driving with 5V but you can drive open drain outputs like this.
Then there is the block diagram on page 4 showing open drain outputs. The equivalent output circuits miss out the output pins altogether. Another clue is the timing diagrams on page 12 only show the output pin states as on and off, not high and low.
The test circuit on page 13 has an arrow going into the chip Iol indicating it is sinking current. Also on the same page the AC characteristics show it being connected to a positive supply (battery).
This chip is much the same as the TPIC6B595 which has a better data sheet.
Look at the application note:-
http://eu.st.com/stonline/products/literature/an/11313.pdf
page 13 talks about constant sink current, and the application circuits are all sink designs.
while the shift registers send out the HIGH signals.
So that explains why it will not work.