Speed, reliability, and how much they interfere with the other things your program is doing.
When SoftwareSerial detects that serial data is incoming, it blocks until the whole byte arrives. HardwareSerial only blocks until the bit arrives.
Depending on what else your program is doing, that may, or may not, make a difference.