Son, I have been using lead based solders for 50 years with no ill effects, I've had my blood checked several times for poisoning for Lead, Mercury and Cadmium among others and has slight levels of lead and mercury...
Well I'm glad that you haven't had any issues, and I know there must be millions of people with similar histories working with lead solder that never got lead poisoning in any form. I have used 60/40 solder for over a decade before any real alternatives were available. It's a well established fact that lead solder can be worked with safely if proper precautions are taken.
However, as you mentioned, there should also be concern about what happens after the useful life of a device. That is the real reason I choose not to use lead solder. Even if I don't dispose of anything I've soldered (not a realistic situation), I'm not going to live forever so eventually someone is going to throw it away and then it will most likely end-up in a landfill somewhere. Therefore, if there's no lead in the solder then there is no lead to possibily leech into the surrounding soil and groundwater. I know you stated that the possibilities of that are low, but a non-existent chance is lower still.
As for the new types of flux for lead-free solder, I'll agree that phosphorus has it's own problems. However, as mentioned I always have adequate ventilation. As far as long term enviornmental and health concerns, it's much less of a problem than lead as there are many biologically useful molecules incorporating phosphorus
so the potential is high for even harmful compounds to be metabolized into something innocous by microbial activity (which isn't the case for lead). In addition I avoid it whenever feasible; and if for some reason a MSDS for the solder and/or flux is unavailable I just take my money elsewhere.
I'm not making any personal judgements about anyone else if they continue to use lead solder. I was just mentioning why I decided to stop personal use of lead solder and, for me at least, the transition wasn't a significant issue.