IMO, a new API call should have been created to drain the tx buffer rather
than change the behavior of the exiting flush() api function.
I would (almost) agree. The basing on the Stream class, though, dictates that the flush() function operate in the same fashion as flush() on any stream. Typically, flushing a stream means that a send of the buffered data is forced, rather than waiting for the buffer to fill. This is now the behavior of flush() in the HardwareSerial class. The unfortunate thing is that there was already a function of that name in the HardwareSerial class, even though what it did was not what one typically expects flush to do.
If one looks at a lot of code posted on the forum, the current functionality of the flush() function is what most people expected the old functionality to do.