How do you have the ground tied together? If the ground from the 9V battery is tied to ground of the AA batteries then it would work ok for that. Once you connect USB to the board and power is then supplied from the PC it may not be enough to keep them properly referenced.
This answer is also given from my understanding of your issue... perhaps a wiring diagram would help for me to better understand question if i did not interpret properly.