You couldn't damage anything, there wasn't a circuit so no current could flow.
Well if the servo "wiggles and then stalls and wiggles a little" current is flowing. Current may flow from the servo +v input somehow out the servo control line when the line is low. I know the 9g servos act erratically and get hot when the servo ground is lost.