Well, that would indicate this isn't doing what you want:
brightness = brightness += fadeamount; //add or subtract to the brightness based on fadeamount
if (brightness == 0 || brightness == 255) {
fadeamount = -fadeamount;
}
Try putting a Serial.print and see what's happening to brightness & fadeamount.