As for the stabilizing cap.. There is no need when run 100% battery power. Your batteries are not fluctuating wildly like an AC adapter. The caps that are built in to the Arduino on the 5V rail are more than adequate. You could desolder all of the electrolytics and not notice when running on a battery
True only if you are not using a voltage regulator. The voltage regulator generates noise; the filter caps are required for the voltage regulator, no matter how clean the DC power source.
Do not try uploading a program if the arduino is underpowered (less than 5V). It can corrupt the bootloader such that an ICSP programmer is required to recover.
-j