Arduino Due Powered with USB battery Pack

I am having some trouble when trying to run my Due off of a 5V USB battery pack (The kind you would use to charge your phone on the go). I try powering the Arduino through the programming port USB but When I plug it in it seems to clear the storage on the Due and I am forced to reflash the program through my computer. When I run it off the USB from my computer everything works fine but then it just doesn't work when I try with the battery pack.

Any Ideas?