It would still be possible to get rid of the de facto standard. Of course any new board has to be compatible to old shields.
The solution is to just provide the second row. For manufactured shields the weird spacing is no issue. Anyone who wants to do a homebrew shield on stripboard is most probably capable to handle a soldering iron. So if the second row would be in place it would be pretty simple to resolder as needed.
It would be even possible to switch by offering boards with both rows for 3 years and bring out a roadmap that tells that in 3 years from now the new spacing will be the official one.
Udo