Strange behavior from voltage regulator

I wouldn't call that 'connected'. It's basically floating in space, in general proximity of where it should be connected. Just lying two loose pins next to each other is NOT a good way of making a contact...

No, you put a crocodile clamp onto the GND lead allowing for a better contact and hence a better measurement...