Just for interest, 5 hours after I started the sketch is still running. The power is down to 3.9V.
So as a proof of concept, it appears that a couple of ultracapacitors can keep your sketch running for 5 hours. This particular sketch spends all its time lighting up LEDs, which is probably a bit of a torture test, comparing to monitoring temperatures etc.
The two capacitors I used cost me $US 32 (for both).
Whilst they take a while to charge, I think this is acceptable. After all, you put charged batteries in as a battery backup don't you, not discharged ones. Either you could organize a more sophisticated charging circuit, or just pre-charge them.
Something like this could keep a security system running for hours, even during a power outage. And if you double the number of capacitors, you double the time they will keep running. Coupled with the suggested power board, with its DC to DC converter, you could probably squeeze something like 8 or more hours of performance out of them (or 16 if you use 4 capacitors).
The question I would have is how does this compare to a more traditional back-up using batteries as far as initial costs, volume (size) used, and recharge time, etc. Most batteries have a pretty flat voltage discharge curve while the super cap would need some kind of boost voltage regulator to match battery voltage Vs time performance, which might be critical to some applications such as using analog input values where a changing reference voltage would hurt calibration accuracy.
So while super caps are interesting I haven't yet seem an application where a properly sized battery doesn't perform as well or better for less costs and space requirements. Super caps have been around for quite a while now, but I've seen few pratical applications for them, and I don't count the auto stereo installations where I think the visual appeal is more important then what they actual add to the systems performance. But then maybe I'm being too pessimistic? Show me the math.

Lefty