There is something called parasitic power; if you don't power a device (IC, LCD) but the controller sets pins high (e.g. 328 continuously writing to display) on that device, your device might still be powered (through protection diodes) drawing current from the controller.
Be aware of that. It's not my area of expertise but setting all pins connected to the LCD to input might prevent that from happening; other's might climb in correcting me if necessary or advising further.