Arduino can output 120 mA or so for one port.
According to the specs the Arduino can output a max of 40 mA on an I/O port - not 120 mA.
What came to my mind: to keep the setup simpler and less power consuming this could be an alternative:
If you do not want to display the R G B color values all the time you could consider to only use 1x 3-digit display and only display the color value of the changing color - let the code determine that by monitoring your potis (of course this only makes sense if you do not want to change colors simultaneously).
Just my 2 cents...