Then I unplugged and switched the wires from the 5v on the Arduino to pin 9 on my ULN2004A, and left Vin attached to the 50k trimpot and Unipolar 2-phase stepper(4vdc, 1.2A/phase) Com A and Com B.
That's what you did wrong - that put all the power of the inductive spikes from the motor through the Arduino board and put large spikes on the 5V line (upto 12V perhaps), and for some reason the 328 survived (luck plus the decoupling caps!).
You must connect the motor supply directly to the pin 9 of the ULN2003. Its essential to preventing destructive inductive voltage spikes.