Ahaaaa, you just want to have a single version on a stick? Then just download the non-install and extract it to the stick. Make a folder called "portable" in there and start the IDE. Now it will probably make a folder called "sketchbook" in that folder. Close the IDE and now you can copy all your project to that sketchbook folder. Aka, just copy past everything from your current sketchbook (by default c:\Users[user]\docuements\Arduino) to that folder. If you installed the libraries correctly (aka NOT in c:\program files\Arduino\libraries) then they will be installed in the portable USB stick version as well. Now you can de-install the Arduino IDE from your computer and you can delete the Arduino folder in your documents.
No need for backup, you're not really installing anything. And even if you where, a install of the IDE does nothing with your sketchbook). But then again, backup NEVER hurts 