The limiting factor seems to be not so much how many matrices can you control but how fast can the arduino update them. As others have said you can use shift registers to control nearly unlimited outputs. If you want to display an arbitrary image you need to row scan your matrices and if you don't update the rows fast enough you get horrible flickering effects.
Just to point this out, but putting the row scanning (or column scanning) on the Arduino is a decision that you make. There are drivers such as the MAX7219 that do the row scanning themselves. You give it the data, it keeps refreshing the matrix at 100Hz or so until you tell it otherwise - it is both the high side and low side driver. So it acts basically like a coprocessor for you. Other chips like shift registers and the TLC5940 make you do that lifting because they are only low side drivers, but have other advantages (mainly cost for shift registers, great PWM capability for the TLC5940).
The limit with the MAX7219 comes if you want to update your image quickly - animations, scrolling, etc. Then you might get artifacts with larger displays because the Arduino can't update all the MAX7219s fast enough. But for static or mostly static content, you can really go nuts with those drivers.
AMS AS1100, AS1107, and AS1130 (charlieplexing) are also high-and-low-side drivers (i.e. coprocessors).
Are there any other good high-and-low-side drivers out there? These drivers make it simple.