hi
how is the pot wired-- is one side of the fixed resistance going to ground? Sounds like the motor driver should work fine at 5V... perhaps the problems you are experiencing are just from the unstable landscape of the breadboard.. perhaps it is time to wire it up and eliminate problems one by one?