The VO pin needs to be about 0.4V on most 5V displays.
Many have a 10k pullup on the pcb. So all you need is a resistor 0R .. 470R between V) and GND to produce the contrast voltage.
Your pot is providing a "variable resistor" if you have disconnected one end terminal of the pot.
Measure the successful resistance. I bet that it will be between 0R and 500R.
Bill Perry will probably know whether panels exist without the pullup. Obviously those displays need to have a pot instead of a single resistance.
David.