Assuming that you'll have "something" to change the brightness with, one more control, e.g. a rotary encoder is enough to change along a "spectrum".
Instead of the usual RGB color space use the HSV (Hue, Saturation, Value) model. Changing the Hue would change the color and Value the brightness. You'd have to clamp Saturation.
You can find a nice explanation and the code to change from RGB to USV here: