The frequency slows down continously but takes time to spped up!

Henri, what board and IDE are you using ?

I use Arduino Due and IDE is 1.5.2.