I don't fully understand some of the more subtle grounding effects, but I figure what's happening is this:
With USB, your ground reference is the actual Earth -- through the PC, mains wiring, and breaker panel at your house. When battery powered, ground is floating. (From a battery, all that matters is the difference between the reference and the V+ supply.)
So, when you touch a Gnd lead, you equalize grounds with the battery-operated circuit, but you're more likely a source of noise for a mains-powered circuit. After all, you're a lower resistance to Earth than the surrounding air (battery), but not lower than copper cable (USB).
I think. Someone might need to put me in my place.