The wires are fine, but ahh, I never connected the grounds. When I was powering both the arduino and the servo with the PC, I guess the grounds were connected internally to the CPU. Now that i'm using two different sources, I have to connect them externally? Just by my (limited) knowledge of electronics, I don't see why it would matter, but thanks. I'll try it now and report back.
I'm guessing the differing performance depending on the wall wart was just a fluke.