Standard DIP's are still made... for convenience only.
DIP packaging (as stated earlier) was developed in the late 60's. I actually worked as a through-hole Wave Solder machine operator in the 1970's and know that in many cases.. DIP chips (and other components) were hand inserted unless you were a BIG company and could afford the robotic stuffers.
The Benefit of DIPS was clear... they were tough and easily hand inserted. Could be easily and cheaply socketed and went directly from R&D to Manufacturing. Micro-miniature cases (similar to SMD's) were created... but they were seldom justified when designs were for standard TTL data systems... (IE Honking Big Computers) which were all the rage in the 70's and 80's.
The reason for the SIZE is simple. You need a certain SIZE pin to withstand insertion force. Anything smaller and you destroy the device if inserting into a 70's era socket.
The design of the internal contents of the DIP are important... that wasted space you imagine is actually where the pins run... all to the center of the package. The pins on DIPS are created by STAMP CUTTING from a singe flat sheet of metal. The pins are radially cut and really do need to use all the space so the 0.100 spacing can be maintained.