You could switch on the MCU and then use the pins to control the keep / switch off the circuit. Something like the circuits discussed in this thread:
The switch turns the MCU on. Then with GPIO2 you check when the button is pressed again and with GPIO1 you switch it off.
There are other variants in the thread. It shouldn't draw not much current when off.
It needs still a few components. To make it small the best is a PCB, it's very cheap and easy to do. Or pack everything very close in a perfboard, using also smd components.