it can be done but it wouldn't last long seems to be the overall jist of it
At least not in the "standard" configuration; I'm sure that if you built a standalone circuit, modded things so that the ATMega was running at the lowest possible speed (internal clock?), and set things up so that it was in "sleep" mode most of the time - you could probably easily run the processor (and maybe a sensor or two) from coin cells.
If you needed more current, there's always the possibility of running off small "specialty" batteries (Radio Shack at one time carried all kinds of "weird" size batteries and cells - they might still thru their catalog). Something like "N" cells, or sub-AA, small things like that. At a certain point, though, 3.3V LiPo batteries start looking more attractive (plus hunting down holders for some of those weird batteries can be a chore).