I watched your video three times and I'm not sure what the issue is. Are you saying the lights on the right should not be fading when the ones on the left come on?
A schematic would really help here. There's lots of these LED matrix projects out there, and they are slightly different - maybe a link to the project you are doing, if it shows the wiring? Then, we should check your actual wiring and make sure it matches up. If the project is posted on the web somewhere, please post the links.
Is it this? LED Matrix - Dual Color - Medium - COM-00682 - SparkFun Electronics