Well, I've decided to scale back my design a bit. 
I started working on a schematic in Fritzing:
http://raccoonrocket.com/gb/egbschematic.png
But after getting it around 75% complete, I noticed a serious problem. Focusing on all the little details, I'd missed a big one and miscalculated the number of pins I need. In addition to the 18 I needed for the array, I also needed one for a piezo speaker, one for a switch, and one for a potentiometer. I'd forgotten the potentiometer. In addition, I'd decided the other day that I wanted to be able to pulse the brightness of two additional leds which would be on the ends of some antennas. That brings the total number of pins I'd need up to 22.
I'm aware that I could use a shift register to reduce the number of pins I need by 7 or so, but at this point it's just getting too complicated. Three chips to gain an extra two rows was already getting ridiculous, four is simply too much.
So, rather than have four 20 led bargraphs, broken up into eight sets of 10 leds, I am instead going to leave some of the leds unused and have four 16 led bargraphs, broken up into eight sets of 8 leds.
Through each of the 8 leds in a column, I will put 20mA, and I will run them on a 1/8th durty cycle with a 70hz or greater display update. I have already tested this out with a single bargraph being blanked 1/8th of the time, and while the display is not blindingly bright it is quite acceptable.
This will leave me with only 40mA to go before I exceed the 200mA limit, but I think I can power a piezo and a couple more leds off that, especially since the other leds should be really bright. There's a couple more leds on my illuminated pushbuttons which need to be lit as well, but thise I can run directly from the 9v battery and so won't contribute to the 200mA limit.
On the upside, limiting my current to 200mA means I'll more than double the run time I'd calculated that I'd get from two 9v batteries, so I should now get over 6 hours out of them. And it may in fact be more like 12, considering that less than half the display will be lit most of the time.
And my wiring will be much simpler as well, for now I will need only the 2803 to sink the current of each column. No more dual 2981's to source current. It'll still be a pain in the ass to solder ribbon cable to all those bargraphs to make my array, but it'll be slightly less of a pain in the ass now. Also, I no longer need to worry about whether the voltage drop between the two ICs the the leds will be too great with just 5v drving them.
Seems like a fair trade for 16 leds.