Great, I have a week's vacation coming, I will know how to spend time 
I just found this reference, from Spence himself:
PB5 when set to work as I/O pin works fine as an input, but as an output, it's' current is extremely limited. Basically all it can do is drive the input to a digital pin on something else...
"Very limited" is not quantified but I guess not enough to drive a LED. But at least it will I think be able to grive the gate of a mosfet that will drive the led. I guess?
And this without resetting RSTDSBL, so still able to use ISP programming from what I think to understand from his comments in that link?