To get the Arduino libraries working on another chip somebody has to port the code across, this is a job that can range from easy to major project.
As the SAM4 is very similar to the SAM3 the job should fall towards the easy end, but someone still has to do it and there are bound to be issues.
Actually the fact that the CPU is an ARM is (almost) irrelevant at the Arduino level as 99% of the code deals with the peripherals, the actual core CPU is hardly ever referenced by the code.
what I need to do to adopt code for the Arduino Due to other microcontrollers
Grab the source code for all the core libraries and get to work...good luck, it's doable but probably not by you if you are asking
Alternatively buy one of the many boards around that already have this work done, for example the Teensy 3.