When connecting the UNO to your Colorduino, you are using the UNO's serial connection to program the Colorduino.
That is why you need to remove the 328 from the UNO or keep the UNO in reset while you are using the UNO's serial connection to program the Colorduino.
From what I could find on the internet, it looks like the Colorduino is programmed with the Duemilanove bootloader, so you need to select Duemilanove as the board type in the IDE when uploading to the Colorduino.
Refer to these instructions for further help on the proper connections between the UNO and the Colorduino:
http://www.instructables.com/id/Lampduino-an-8x8-RGB-Floor-Lamp/step16/Programming-the-Colorduino/Since the Leonardo uses a different type of USB serial interface, you will not be able to use the Leonardo to program you Colorduino.