but others corrected me that technically the standard tone library is only getting 2.5 volts because it's being pulsed half the time.
The others are wrong.
There is no need to get together. PWM is a 0 to 5V signal. At 50% it is still 0 to 5V, it is never 2.5V.
However the average voltage is 2.5V but the peak to peak voltage is 5V.
So your push pull system produces a +5 to -5V signal as seen by the speaker. That is a 10V signal. It has an average voltage of 5V but the problem is that an average voltage tells you nothing. Suppose you use a 25% duty cycle on your system. The average voltage is 2.5V but it is still a 10V peak to peak signal. A 25% duty cycle will sound as loud as a 50% duty cycle tone. The only change a duty cycle makes is to add or subtract a DC offset to the signal and you can't here DC.
When measuring loudness of an electrical audio signal there are several ways you can express it.
Peak
Peak to peak
RMS
Avrage
and peak music
They all give different answers.