The ShiftPWM library will let you control 768 LED's (256R+256G+256*B) with 32 levels of brightness and refresh fast enough to not cause major flicker.
http://www.elcojacobs.com/shiftpwm/
If you want a non-matrixed display, the SPI interface on a 16 MHz Arduino can pump data at 8 million bits per second (1 million bytes per second). If you want a frame rate of 100 frames per second that gets you 10,000 bytes. If you use one of the LED driver chips that has 16 bits per LED that would allow 5,000 individual LED's (1666 each of Red, Green, and Blue). Each chip controls 16 LED's so you'd need 300 of the driver chips and they are not cheap.