The problem with your simplified circuit is that when the Arduino is powered down, the pin driving the second P-channel mosfet will go low because of the pin protection diode, and this will turn the mosfet back on. To fix this, use an NPN transistor or N-channel mosfet to invert the signal, so that you need a high on the pin to keep the P-channel mosfet on.
You can leave out the first P-channel mosfet if you connect the push button across the source and drain of the second P-channel mosfet, so that power flows through the push button initially. Or you can use the pushbutton to turn the second P-channel mosfet on, bypassing the NPN transistor.
An alternative solution for a final build is to use a barebones Arduino without the FTDI chip so that it consumes very little power in sleep mode. A standard 16x2 text LCD can be powered from a digital output pin through a 100 ohm resistor, with the backlight controlled from another pin, via a transistor if necessary.