Many projects do not require a specific voltage. The ATmega chip in the Arduino can run on any voltage between 1.8 and 5.5 V, although at lower CPU clock frequency with low voltages.
Here a linear regulator perhaps makes more sense since the power loss is small and a
simple switcher might risk discharging the battery flat (dangerous). You can get chips
that act as regulators and LiPo management all in one, these are surface mount invariably
though.
You need a very low-drop-out regulator - this means reading the datasheets and finding
one with 0.3V or less (most are 0.5 to > 1.0V - marketers use terms like LDO very loosely)
Nearly all 3.3V devices are rated for 3.0 to 3.6V, and freshly charged LiPo batterys generate
4.2V or more, so certainly not compatible without a regulator. The regulator should have
automatic over-current and overheating shutdown - very important with LiPo as some
form of fuse / safety cutout is necessary due to the vast power such batteries can source.