Hi,
The actual instantaneous current will be a bit less than 20mA because the transistors or other drivers on the columns and the rows will "drop" some of the 5V, so the voltage accross the LED + resistor could for example be as much as 1.5V less than the full 5V. This is is still true even if the Arduino outputs themselves are driving the matrix directly. Less voltage means less current, of course, but you could adjust the resistors to a lower value to compensate (as long as there is still enough voltage to drive your LEDs: blue and white LEDs can need up to 3.2V to run at all).
However, the main reason why LEDs are dimmer in a matrix is because of the multiplexing. It may appear that all LEDs are on at the same time, but if you had a slow motion video camera, you could see that they actually switch on and off in groups. Usually the groups consist of either all the LEDs in a row or all the LEDs in a column. For example in an 8x8 matrix, only 8 LEDs might be on at any instant. If they receive 20mA when they are on, they are receiving 20/8 = 2.5mA on average.
This may sound like it makes the display very dim, but there 2 reasons why it might not be. First, the human eye is not "linear" in how it perceives brightness. 20mA will not look twice as bright as 10mA, it will only look a little brighter. Secondly, LEDs can be "over-driven" to compensate for the multiplexing to some degree. If an LED is rated at 20mA maximum continuous current, it may be able to take 80 - 100 mA for short periods. You need to check the data sheet for the LED to find out how much and how long. Usually I have found its OK to use double the max continuous current, if the LEDs will be on for only one eighth of the time.
There is a danger to over-driving LEDs. If the multiplexing relies on your sketch running correctly (rather than using an external chip like max7219 to perform the multiplexing), then if your sketch stops for some reason, the LEDs that were switched on at the time will then receive more than their max current for a long time and could be damaged.
Paul