However upon connecting the power to the circuit, the voltages on the battery and the output of the regulator drop quite a bit. Is this simply because the battery can't source enough current to the circuit?
That is correct. The battery has to be selected for the maximum load current it is being asked to supply. Also how long you wish to battery to be able to supply that load current is another selection criteria you must make.
Lefty