An Arduino will run off (eg.) 4V connected to the "5V input" pin (ie. bypass the regulator)
I wouldn't recommend this with latest versions of Arduino.
The regulator is sensitive to being reverse driven.
Add a diode from 5V (anode) to Vin (cathode) to prevent damaging the regulator.