So before I destroy all the matrix, could somebody tell me what's i'm doing wrong?
Basically no, because what you are doing sounds sort of OK.
The bit that is not is that 3V3 from the arduino is limited to 50mA on many models of arduino.
So you are either not doing what you think you are doing, doing something else you are not telling us or have a very low current capable matrix.
I did get some RGB LEDs from ebay and I found that about three of the batch of 50 burnt out with only a few mA going through them. I also found that the red was rather dim on all but one of the LEDs. I think that some LEDs you get on ebay are actually from the reject bin.