Thanks for the answer !
Well maybe I was wrong on that point, but the thing is that I tried again today (with the diode being at the right place !) and guess what...
Now the motor never spins... =( (I mean in the circuit, if I plug it only in circuit with the 9V battery, it still works)
Actually I think that at my previous attempt, I inverted the battery snap poles, plugging the (-) one to the battery's (+), and vice versa.
It may explain why it was spinning before, and why it isn't now.
The thing is that when I press the button, it still doesn't spin...
I begin to wonder if, during my many manipulations trying to get it to work, I haven't broken something like the transistor.
Especially if, as I think, the current has been reversed at any time.
(If that's it, given that I tried with the two transistors bundled with the kit, they may be both broken 
)
By the way this may seem very silly, but, following the book's circuit schema, when they say the metal tab to be "facing away from you", they of course mean that the face with the little black rectangle thing must be facing the Arduino, right ?
I must admit I'm running out of ideas ... 
EDIT: After having changed all the pieces of the circuit (push button, transistor, resistor ...) I finally got it to work as expected ! This way, I'm not afraid anymore to have something broken since I know it worked at least once. Thanks for the help ! 