Using an emergency cell charger as a battery pack?

I am trying to use the guts of an emergency cell phone charger as a battery pack for an Arduino. It's sold under the name "Digipower Jump Start", RadioShack has them for $13. It has a mini-USB port to charge it, a mini-usb plug to plug it into your cell, and a 220mAh Li-ion battery.

To connect it to the Arduino I connected its ground to the Arduino gnd pin. Then connected the 5v pin of the Arduino to the center of a DPDT switch. One side of the switch (let's call it "off/chg") goes to the USB-in 5v pin on the charger and the other side of the switch ("on") goes to the USB-out 5v pin on the charger.

It works just fine in "on" mode, the Arduino is powered by the battery.

In "off/chg" mode it half works. It will charge the battery when a USB cable is attached to the Arduino, which is what I wanted.

The problem is that if I unplug the USB cable, the charger "leaks" about 3.5v from it's USB-in port, not what I wanted/expected. This is enough to power some of the LEDs and of course drain the battery.

Is there a way to stop this leak other than finding a nice little on-off-on (on-off-chg in this case) switch?