Yes all these chips are pretty complicated. 1465 pages for the SAM and 448 for the Mega328. Most of the stuff you are used to like UARTs etc aren't that much more complicated, but then you have the PLL, NVIC, DMAC etc etc on top. Truth is 98% of people don't need them so at the end of the day it's not as bad as it could be.
If you have a proper framework this stuff is either disabled or setup well enough to get started. There are gotchas though, on the LPCs the GPIO ports don't work until you enable their clock, I spent some time wondering why my writes had no affect. Same goes with the other peripherals.
Presumably this sort of thing will not bother any Due users.
Rob