Why am I getting negative numbers?

You need to use the long type for your redtotal/greentotal/bluetotal variables, they are overflowing.

The output of analogRead is in the range 0..1023 inclusive and cannot be negative anyway.