Yes, it's worth it, if you want long battery life.
But for an intermittent use like you're describing, the biggest savings comes from the complete powerdown mode where you've shut off all the LEDs. That is the one you want to focus your efforts on, usually with a current meter connected. If the brownout detector is enabled but everything else if stopped, you should have about 30 uA current. It's very easy to find something else besides the processor is using a lot of current. For example, a 7805 voltage regulator uses 4 mA all the time. Running directly from the battery without a voltage regulator saves that!
When you are running, keeping the current low helps if you spend a substantial amount of time in on mode. But also a big improvement comes from being able to run on a lower voltage. Especially for alkaline batteries, the voltage declines gradually as the batteries die, from 1.5 volts to 1.0 volt. So if you can still run "normally" on 2 volts, you'll get a lot more life out of the batteries than if you can't run when they fall below 1.3.
Probably a difficulty you'll have running down to lower voltages is being able to turn those LEDs on. Red LEDs usually need 1.8 volts minimum, green about 2.1 volts.
So to put some numbers on all this, first a AA battery has about 2500 mAH capacity. That's a rough guess, maybe it's more if you run all the way down to 1.0 volt per cell. Suppose while you're running, current consumed is 30 mA. That's enough to run on about half of those LEDs with 4 to 5 mA and run the processor. Suppose power down current is 0.1 mA, which is not too hard if you've carefully checked nothing else is hogging current when you're in full stop mode.
Suppose your board gets used 4 times a day, each time staying at 30 mA for an hour. That's 120 mAH of the battery. The powerdown runs the other 20 hours at 0.1 mA, consuming 2 mAH of the battery. Using 122 per day, you should last about 20 days.
In this case, using less current in the LEDs or shortening that powerdown timeout (assuming the user isn't actually using it for all 4 hours) buys you a lot, since active mode is burning 122 mAH every day.
It's also pretty easy for the powerdown mode to burn too much. I just picked 0.1 mA, which is doable if you're careful. But look in another thread where someone reported their board was about 9 mA in powerdown mode (clearly something else on the board was burning the current). That would put the powerdown at 180 mAH per day.
This simple math assumes, of course, the current is always constant in both modes, which is usually approximately the case.