How about your common grounds. Is the grounds connected when using battery, or just when using USB ?
I don't use a battery, I'm using a 200mA 9V power supply, I just connected GND to ground but I think I also tried connecting it to GND1 also.
The 5V coming through USB is filtered. Though the 9V should not be noisy enough to create what you are describing.
You can try adding 0.1uF and/or 4.7uF caps between Pos and Neg close to the battery.
This also applies to a power supply?
If it's Chinese made, anything's possible. If you have some caps lying around, throw some in and see if it makes a difference.
Describe the "noise" you are hearing. High or low frequencies?