Perhaps you have not selected the correct board type, or the boards entry that you have selected is not configured correctly. Did the documentation with the Colorduino board tell you whether it needed any non-standard settings?
putskiz:
I had to remove the Atmega from the arduino Uno board, not very convenient.
Next step is to try with USBtinyISP instead of arduino.
You might be better buying a USB2Serial device like this to program it else you may find you have to remove the LED matrix to get access to the ICSP header every time you want to program.