Let's talk timer5 setup a little...
To fire an interrupt every 100us I would recommend CTC mode (Clear-Timer on Compare) using the Output Compare registers. See datasheet section 17.9.2. You then need to choose a prescaler value and an OCR value that translates to 100us.
For a 16MHz system clock, 100us = 1600 clock ticks.
There are some options but you've used prescaler = 1, so we'll continue with that.
I think you understand these concepts, you just seem to have selected some incorrect values for your registers.
This should be the register setup you want (untested)...
// this code was wrong when first posted. See following posts...
Note that the code you posted sets up Timer5 in PWM, Phase & Freq Correct mode which uses ICR5 for the top value. I'm not familiar enough with this mode to be able to predict/explain how it behaves, but it definitely won't be what you want.
-Mitch