In the datasheet for the Atmega2560:
You can see that for register TCCR3A the "bit" called COM3B1 is bit 5 (the 6th one across, it is bit 5 if you start counting at zero).
So your expression: 1 << 5
... gives you that particular bit. If you "or" that into the register you have set that bit.