I've put 1.8V on PWM specially to reduce the base-emitter current,
No you haven't, you still have the full 5V unless you smooth the PWM signal which you don't want to. If you put less base current the transistor will not turn on fully and so will drop power in the transistor.
So with the motor loaded I only let 500mA run through the transistor.. Not 6A or even 3A. Where am I wrong
It gets hot because of the power not the current. The power is voltage times current. So even at less than the maximum current the voltage may be such that the power rating of the transistor is exceeded. Even at the minimum voltage (Collector emitter saturation voltage) you can't run the maximum current because the power would be too high. In a continuous mode of operation it is the power that limits the maximum usable current on this device.
See these sites:-
http://www.thebox.myzen.co.uk/Tutorial/Power.html