am I merely lucky?

is the chip itself the only thing at risk,

Mainly although the USB / serial bridge can fail as well by doing things like taking too much current from the 3v3 line. Also people have blown up their USB ports on their computers as the degree of protection is not perfect.