0->255: Green fades from 0 to 255 while RED stays at 255.
255->512: Red fades from 255 to 0 while Green stays at 255.
512->786: Blue fades from 0 to 255 while Green stays at 255.
768->1023: Green fades from 255 to 0 while Blue stays at 255.
I think you'll have to figure-out what you want to do first... What do you want to see at minimum? In the middle, at maximum? Etc.?
You'll probably need some if-statements to handle different ranges of the pot differently.
With the 10-bit ADC you can only get 1023 different color combinations.
As you may know, 0-255 can is represented by 8-bits. So a single 24-bit variable (which requires a type-long) can represent any possible color & brightness combination.
Then you do bit manipulation to "extract" the groups of 8-bits for the 3 different colors.
With 24-bits you can count to (you might want to double-check that) (1) so you could simply map 0-1023 to 0- 16,777,216. But of course, you're only going to get 1023 different colors with most values being skipped-over.
And with this "simple" approach, you're going to get discontinuities... When a group of bytes counts 255+1, the number rolls-over and 255 suddenly becomes zero.
(1) This becomes easier if you can think and program in hexadecimal. With 8-bits you can count to FF in hex. Each byte is represented by exactly 2 hex digits. With 2-bytes we can count to FFFF and with 24-bits we can count to FFFFFF. With RGB values each group of 2 hex digits represents a color so in hex we can "see" each color without any math.