Almost any output from the Due can tolerate delivering at most 3 ma,
or a 1100 ohm load resistor.
Typical 8- or 16-ohm speakers, or 50-ohm headphones will essentially
"short out" the Due's output pin, and probably destroy your Due.
A typical (maybe not all) amplifier's line-in input has an input
impedance of something like 10,000 ohms (10k), so the would
appear to be (barring other issues, like grounding, etc.) a better match.
Analogy: What happens if you connect a speaker to the 12v battery
in your car? (typically, the speaker explodes)
So, when things are connected, there must be compatible, otherwise
one or the other, or both can be severely damaged.