Understanding the components of a LED matrix

That is quite a helpful tutorial! However I still don't understand how you'd only ever source or sink 1 led as a worst case. Wouldn't the worst case be that each chip could be handling 8 LEDs, just one is per pin, and the other is a single pin? Take the last line in the row-scan example: each LED in the row is lit. That's 8x20mA sourced on a single pin (the walking bit), and 20mA sinking on each column. If those were connected straight to the arduino, wouldn't that be a dangerous 160mA? I feel like I'm misunderstanding some fundamental step here :~

This will be a single color matrix with no dimming for clarification.