USB Interface / Driver issues with Arduino DUE compatible design Using ATSAM3X8E

Sorry, I am not a USB person.
Try comparing the schematics. Maybe you are missing an important part.