Think of it as power always coming from the battery, but the battery is always being charged whenever external power is present.
The LiPo Rider is a bit cheaper and also has a low voltage disconnect for the battery to prevent overdischarge. It also takes a few weeks to ship from Seeedstudio, unfortunately.
Also take a look at the LM1117 datasheet from TI for a battery backup scheme.