I did one little breadboard project with a 3.3V GPS device, the only 3.3V device I have interfaced so far. I did it with an ATMega328P so I didn't have 3.3V from an Arduino.
As for power, I used this regulator which seems to have a good reputation, works fine, multimeter says it is spot on:
Now the nice thing is that the GPS could be interfaced to the INPUTs of the ATMega328P fine - 3.3V drives the inputs without any trouble, and the GPS itself needs no input, it starts sending data once it has it. So I was set there.
If you need to send output to a 3.3V device you need to use a level converter like the one mentioned above which seems to be bidirectional and intelligent and out of stock. As an alternate, there is this not quite as elegant solution:
But it is now out of stock too. What gives with all these level converters out of stock?
Sometimes the better choice can be to run the whole system on 3.3V. Depends on the sensors and other devices involved.
When running on batteries, an Arduino is often not a great choice. For example, an Uno board will consume 3-4 times the current/power of just the ATmega328P microcontroller alone. That is a not criticism; after all, it is a development board, and that is the price paid for the convenience it provides. While the 328P can be put to sleep, and that is a great option for battery-powered applications, certain components on the Arduino board continue to draw power regardless. I would strongly consider a custom circuit design for a battery-powered application.