1) Do not connect power supplies anywhere other than the barrel connector, unless you have a good idea what you're doing.
1a) This means being extra careful if you connect an externally powered device (like a servo motor) to the IO pins.
2) Be careful of the power supply that you DO connect to the barrel connector. A supply described as "12V 300mA" may sounds like it meets specifications, but can (somewhat legitimately) provide a significantly higher voltages.
3) The AVR IO pins are only capable of supplying a limited amount of power (~40mA at 5V - 200mW) Avoid connecting loads that would exceed this rating, such as LEDs with no current-limiting resistor, lamps, motors, solenoids, etc.
4) In addition to the limit on each pin, there is an overall limit for the AVR chip that is lower than the pin limit times the number of pins.
You can not put a 40mA LED on each of the 13 digital outputs.
5) Be especially careful with inductive devices (motors, relays, solenoids.)
6) Avoid static electricity. Winter is a lousy time to get a new Arduino!
7) Nothing on the Arduino board should be getting too hot to touch. If it is, it probably means you've violated one of the above and you should stop and rethink your circuits even if things appear to be working.
Avoid putting the arduino on top of conductive things (coins or wires) that might be littering your desk.
9) It is better not to plug in wires and other devices to the arduino while it is turned on. OTOH, this is a bit hard to resist...
That's all I can think of at the moment. Most of these have other discussions on the forums that go into more detail. Or you can search/ask when a particular item comes up in your experiments.
As electronics go, the Arduino is relatively Robust and relatively cheap. The risk is lower than adding RAM to your desktop computer. But since the Arduino is an exposed experimental platform, the risk present more often.