I think the 'normal' wiring is to have the load (motor) wired between the collector and the power source. Also you will need to put a snubber diode across the motor to stop flyback current destroying the transistor. e.g. a 1n4001http://en.wikipedia.org/wiki/Flyback_diode
The phototransistor may not be allowing enough current to flow through the base to enable enough current for the motor. Replace it with a, say, 10K potentiometer and try varying the speed with that. Make sure you don't go down to zero ohms on the base though. You may want to put a 1K resistor in series with the pot.