After some further research, I found this page: Gammon Forum : Electronics : Microprocessors : Interrupts
In here the author measures the time needed to enter and leave ISRs ( 2.625 uS for both ). I plan on doing frequencies of 325KHz (3uS period) so that would leave me less than 500nS for the actual ISR code. I really thought that having a clock speed of 16MHz would leave me enough room but I guess not...