How to calibrate RGB LEDs?

Another reason for the unexpected color is that the response of the human eye to LED brightness is not linear.

While luminance changes mostly linearly with LED current brightness response of the human eye is roughly logarithmic. An even better way is the CIE lab brightness correction.
So in essence the human eye is more sensitive to changes in luminance in dark conditions than in bright conditions. On a usual 8 bit scale with 0 being fully off to 255 being fully on a change from 3-8 can clearly be seen but a change from 245-250 cannot.

With the 8-bit PWM limitation on most Arduino boards linearisation is hard to do smoothly.