I agree with Imahilus on that it depends with the group of children. I remember back in high school there were some classes I took that were just full of delinquents and others who actually applied themselves with all sort of Advance Placement courses.
If your classes are consistently composed of 'good' kids, I would say Arduino is safe and fun to use for them to learn. Motors, speakers and IR remotes like what people stated above are agreeably fun to learn to use. I would even consider photocell sensors and maybe thermosats too so they can see some practical applications of the board.
If your class consists of kids who like to stick their gum under the chair, tardy to class and be disruptive, I would just stick to programming on the computer to do simple things like print things on the screen (i.e. "Hello") and ask for inputs from the keyboard. The less things kids can get their hands on and steal, the better for school.
Another suggestion would be to have the students buy some simple electronic kit that way you don't really spend anything and you & the school do not have to worry about vandalism or theft. Just get a head count and a student agreement form saying they will pay for 'blah' if they are to be enrolled in your class. It's realistic for some individuals while others may be over their heads; just got to work with their attention spans.
(The public high school I attended was full of assholes and thieves. Just something to be mindful of)
What skills would I need to pick up in order to use the Arduino effectively?
Considering you have a computer science background, you would not need to learn much other than knowing where to put what. Programming part of arduino is the most challenging aspect for me (c language). The rest, like the circuitry and how to wire motors, lights, etc., is easy. I tend to google (search online) diagrams and schematics and usually find what I am looking for (i.e. bi-polar stepper motor schematic with Quad H-bridge driver). There are certain things to be aware of such as color coding of resistors (just find a legend somewhere online and print it large on a poster board and tack it to the wall in your classroom).
So to put it as simply as I can, you would need to know how to search the web to learn how to build certain circuits. (Using google images is the best)
I can program and I'm not afraid of learning a new language, but I've never touched a soldering iron. Would I need to get a deep background in electrical engineering before I try anything?
So easy, a caveman can do it. Just a few pointers with a soldering iron is 1. don't touch the metal part of it unless you like seared flesh or want to train for tatoo pain 2. Pull solder away from the spot your soldering first THEN the soldering iron
In general, I would recommend using solderless breadboards that way you just plug everything in and completely avoiding the soldering iron issue of someone burning themselves in class. I would recommend that the kids wear safety goggles. A classmate in my Circuits and Instrumentation lab short circuited a 9v battery on accident and it shot straight up a good 2-3 feet busting the casing of the battery and leaving it very hot.
Wikipedia can give you the short versions of functions of different things like resistors, capacitors, etc. if they are in parallel or in series. Stuff like that.
I hope this wasn't too much on your plate for me to tell you. Hope everything works out well.