Should I worry about how I treat the battery?

Are we talking Lithium Ion or LiPo (lithium polymer) here? LiPo are much more dangerous (do not charge unattended, watch out for swelling, always use a protection circuit (some have them built-in, BTW, but its hard to be sure without dissection). Lithium Ion are more stable in the longterm but still its a lithium battery. The safest are LiFePO4 lithium ion cells which are harder to set alight.

So the answer to the subject line is "yes".