It doesn't sound like the LCD is being powered up on it's Vcc Pin (2)... and the device is getting it's power through input protection diodes (Diodes from input pins to BOTH Vcc and Ground)... Pulling D4 should NOT make the LCD Quit working. The LCD is almost CERTAINLY, NOT wired correctly as the LCD will show a black bar where the characters are displayed with pins 1 (Gnd), 2 (Vcc) and 3 (Contrast) connected ONLY. R/S, E, R/W and data lines would NOT change that. It sounds like the contrast pot is wired correctly (Pot Wiper pin to 3, Pot CCW pin to 2 (Vcc) and Pot CW pin to 1 (Gnd). However even if those connections are good on the LCD the LCD Vcc pin Must be connected to +5V (Arduino) and the LCD Gnd Pin Must be connected to the Arduino Ground (All at the Arduino board) FIRST. IMO
Doc