I think all answers are in these 3 links:http://arduino.cc/en/Main/ArduinoBoardDuehttp://arduino.cc/en/Reference/Serialhttp://arduino.cc/en/uploads/Main/arduino-Due-schematic.pdf
But I'm right with you, it's difficult to understand, it's complex schemas.
I've tried some cases and I can confirm that docs are right.
Case of Serial1, Serial2 and Serial3 are simple: The pins 14 to 19 are connected to the AVR and can have other usages.
Case of Serial is more complex: Pins 0 and 1 are connected to the AVR and to the USB programming port. If the USB programming port is in use (when you program the AVR or when you communicate with the AVR), pins 0 and 1 are connected to the programming port and don't be used for any other usages.
Case of SerialUSB is different: No pins are used and you must use the native USB port to communicate with the AVR.
Take care of special RESET/ERASE actions at 1200Bps on the USB programming port and the USB native port.
Take care of NO RESET action at open/close of the USB native port.