In my opinion, this difference is due to interference caused by the TM1637 driver. With DMM, I don't think you'll see them because they're fast and short. If you have an oscilloscope it will help. Connect the wires of the TM1637 driver and the driver itself away from the potentiometer and you can add a small capacitor to the analog pin (ceramic capacitor from the analog pin to ground) connected to the potentiometer to filter these ripples. Maybe that will help.