Sorry, stepped away from my computer for a few...
Digital pots are not isolated at all. If its getting hot, its probably fried. There is no easy way to do this isolated, you need to know more about the motor controller, in particular what voltage it uses on the pot, whether it has open-circuit and closed-circuit failsafe modes on the potentiometer connections. The receive side needs power from the controller circuit too of course.
The voltage is 0-5V. How can I tell if it has failsafe modes? I'm not sure what you mean by "receive side needs power from the controller circuit"?
You've posted no details of the motor controller, this is the primary piece of information required.
I can try and get a schematic from the manufacturer, but that might take a bit.
Can you post a diagram of how you have the Arduino, what model, and the digital pot and motor controller please?
Can you please post a link to data/specs of the motor controller?
I was able to get the digital pot to work with an LED and an external power source, so I don't think it's the digital pot circuit. Once I connected the digi pot circuit to the motor controller, that's when it got hot and acted erratically. It was connected in the same orientation as an analog potentiometer with respect to wiper and 0V and 5V.
Why dos it have to be isolated? That's rather unusual.
When I was sending a PWM signal straight from the Adruino, the only way I could get the motor controller to work is if I also connected the 0V signal from the motor controller to Arduino ground. This worked on its own and controlled the speed well. However, after connecting a display to the Arduino, whenever the motor went from off to on, it would reset the display. That is the why I reckoned I needed to isolate the signals coming from the motor controller.
Not sure if this helps clarify or not? I'll see about obtaining a motor controller schematic.