I think I got it figured out. I was under the assumption the DAC made its own internal reference voltage - I thought I had read that somewhere or maybe I misunderstood. So I wasn't too concerned with that input voltage and never gave it a second thought.
Anyway, I found the use for that floating 3.3V coming from the old potentiometer terminal. I'm now using that to power the DAC VCC, instead of powering it from the ESP32.
Now both directions are working evenly and I notice no stuttering anywhere.