I have a similar RF remote control, based on a promini.
I am running 8MHz from a LiPo battery ~4V.
It goes into power down sleep mode after each key press, a keypress interrupts it out of sleep to read the key, transmit, and go back to sleep.
Draws ~9mA in sleep, ~15mA while sending.
Power LED draws some, battery charging chip draws some, RF transmitter draws some, atmega draws some.
Can you post schematic of your design?
Why 20MHz vs 16MHz?