I'm a bit skeptical about this:
void TimerThree::setPwmDuty(char pin, int duty)
{
unsigned long dutyCycle = pwmPeriod;
dutyCycle *= duty;
dutyCycle >>= 10;
if(pin == 5) OCR3A = dutyCycle;
if(pin == 2) OCR3B = dutyCycle;
if(pin == 3) OCR3C = dutyCycle;
}
First, the multiplication might overflow. Second, after shifting dutyCycle (a 32 bit field) right 10 bits we have 22 bits left. But OCR3B is a 16 bit field. Now maybe the numbers are OK. Can you provide actual figures here? Eg. from debug output?