Arduino Micro VIN/USB Problem

I'm trying to use a wall-wart to power my Arduino Micro so that I don't have to use the USB. I have a stepper motor and one limit switch attached to it. The problem I'm having is that when I just power the Arduino from the 12V wall-wart via the VIN pin, the Arduino (or more precisely the stepper motor, because I tested it) draws 330ma. But, as soon as I plug in the micro USB (with 12V still connected via the VIN), it drops to 40ma. Goes right back to 330ma if I unplug the USB. The stepper motor is connected through a driver, so that's not it. Any idea what's going on?

Please add a SCHEMATIC as it a little unclear how things are actually connected.