SAM3X8E debugging via SWD problem

I have a Segger J-Link connected to the Arduino Due board and I am programming it via SWD with IAR Workbench or Atmel Studio.

It's basically working really good, but I have a problem when I am trying to configure the core clock frequency. Default it's running on 12MHz but I would like to configure it like Arduino does it to run at 84MHz. When I write to all corresponding registers it's all working out fine, but as soon as I switch the core clock to this 84MHz PLLA clock, the debugger just runs wild. It's either stuck at adress 0, or it just runs through the disassembly without omputing any instructions!

I have absolutely no clue what seems to be the issue here?!