OK, first thing is, you need whole numbers (ints) not floats for the PWM outputs, so there is little point in keeping the bit after the '.' and I wouldn't even botehr rounding it, because the colors are just not going to be that accurate.
I don't really have time to write the C for you, but the pseudo code I'd use is as follows.
int r = 0;
char ch1 = Serial.read();
char ch2 = Serial.read();
char ch3 = Serial.read();
r = (ch1 - '0') * 100 + (ch2 - '0') * 10 + (ch3 - '0');
Serial.read(); // skip over unwanted characters to get to the right place for the 3 green digits
// repeat for green and blue.
Anyway, if you get the idea and get it working long hand, then you can always optimise things into loops if you want.