Yes, that is the battery for cigarette lighters and remote control fobs.
For this motor, you would want either a battery carrier for eight alkaline "AA" cells:
or for really serious use, two of these:
Now I am not sure whether the stall current is significantly greater than 400 mA but it says it is a "brushless" motor so it may be limited. Motors tend to have a much higher current draw when stalled or at the moment of starting up.