TIMSK3 = 0x10 ;
is clearly wrong as that bit is read-only.
What mode do you think you are programming? Normal mode 0b0000 is what you
have programmed, and you are setting the clock input to be T5 (pin 47) falling edge,
so you should be seeing the counter incrementing once per falling edge on that
If your pulse source isn't debounced that could explain why you see "random"
counts, but that could also be because you are connected to the wrong pin. T5 is
hard wired as pin 47 (from my notes, haven't actually tested this, can someone
If you are changing the timer mode it is a good idea to change all three
control registers, TCCR5A/B/C, so that you don't inherit state from what it
happened to be before (although you should know what that is as you have
the Arduino sources).
TCNT5 is a 16 bit register, I would use 0x0000
rather than 0x00
to make that obvious in the source code.
You are also setting the input capture edge bit (0x40 in TCCR5B) despite the
fact you are apparently not using input capture.