The regulator is going into thermal limit... small dc motors draw in the 100 to 300 mA range and the load on the regulator is about 2 watts of power, more than enough for it to thermal limit. the regulator will adjust the voltage to the point where it is just at the maximum package temperature listed in the spec... I used to use the smaller versions for thermally stabilizing a cyrstal oscillator, very handy when building accurate frequency counters. crystal, silicone insulated wire and wrapped in a bit of glass wool and stuffed onto a cavity carved from a small styrofoam block and plugged with a larger piece of foam makes an Excellent poor man's Crystal Oven.. The only heat loss is through the power and oscillator leads and If one carefully wraps the resistance wire used to load the regulator around both crystal and regulator it becomes very thermally stable when properly insulated, perhaps the only use for a mistreated voltage regulator.
Bob