Yes you are right! It doesn't work when an external power source is connected.
In src/ASF/sam/boards/arduino_due_x/arduino_due_x.h: - Un-define USB_VBOF_GPIO - You will also have to remove its gpio_configure_pin() call in init.c
This will make it work with an external power source. This is probably not the "correct" way to do this. When I have some more time to play with it I'll look at the schematic and reply back.