I've connected one pin of the pot. to my AREF pin, and the other to an analog pin.
Errr...this doesn't sound right. I'd leave the AREF pin alone. If it's "powered on by USB" then reading the values relative to the Arduino's 5V reference should be good enough.
You also have to be specific regarding which pins of the pot you're hooking up where. Do you have a diagram you could maybe post?
The Gadget Shield
: accelerometer, RGB LED, IR transmit/receive, speaker, microphone, light sensor, potentiometer, pushbuttons