What exactly is the time you are trying to minimise?
You could skip the bit that spits all the values back to the serial port.
I am not sure what this is doing:-
RedVal[i] = Value[i*5] * Value[i*5 + 3] / 255;
GreenVal[i] = Value[(i*5) + 1] * Value[(i*5) + 3] / 255;
BlueVal[i] = Value[(i*5) + 2] * Value[(i*5) + 3] / 255;
but a multiply and then a dived by 255 is time consuming. Why 255? I would have though 256 and then you can do it with a simple shift.
Have you remove the map() function, I spoke about before?