Hi,
Disconnecting the gnd from the regulator does not turn it off.
It just turns it into a resistor for current to continue flowing through it , Vin to Vout and then to gnd through the 328 and the display.
You do not know if the 328 has properly turned off and if on "wakeup" that it is really getting the proper supply voltage increase to trigger a RESET.
Measure Vcc on the 328 when you have it "switched off" to check that there isn't any supply voltage.
Tom.... 