Voltage fluctuating too much. How to fix?

How can I make the 5V deliver 5V so that I don't have to tweak the code every time I change the way it's powered?

As Jack says, run the peripherals from the Arduino 5V pin, not the Vin pin. I don't think Vin is even valid when the USB is active.