@Patduino, dhenry: I thought it was some kind of interference, could you please elaborate more, or perhaps link me to a page where it explains what is going on? I'm not exactly sure what voltage floating is. Also, advice on how to stop this would be appreciated.
Connect a 10k resistor from the base to ground. That'll keep the base voltage 0 when not connected to the Arduino. It's a good idea to keep it there even when it is connected. Voltage floating just means that the base isn't tied directly to any voltage source, so the voltage fluctuates in value, sometimes it has enough to trigger your transistor. The 10k pull-down resistor will tie it to ground so it won't do that. Google pull-down resistor for more. This is a very common problem.
Also, when you connect the Arduino GPIO pin to the base, put a 1k resistor in between them.