My mistake was thinking that since the button press got me in to AT mode at start up I didn't need to keep pin 34 HIGH.
It's an easy "mistake" to make and I made it myself as the "instructions" aren't very clear on this. Just use whatever AT commands you need to use until you come across one that doesn't respond unless you keep the button pressed, and you'll know which ones require this.
Fortunately, you don't actually have to keep the button pressed the whole time to use these "extended" AT commands, only when you have to use them. So you can press it briefly on power-up to get the HC-05 into command mode, and then re-press it and keep it pressed when you need to enter these "extended" commands.
I found it easier to first type the command, then press the button and keep it pressed, then hit enter. A pain, but not too bad given that you generally don't have to enter AT commands often. If you do, or need to enter them programmatically, then you have to look at more permanent solutions, like a switch or soldering a wire to Pin 34, or perhaps building a small breakout board.
I actually built such a board to do level shifting on the Rx pin, so that I could connect it to a 5V Arduino and not have its Tx pin fry the HC-05's Rx pin. It wouldn't be that difficult to connect pin 34 to this board in case I have a need for it.
It's just a small piece of perf board with a 6 pin female angle header and a 6 pin male angle header, with the 4 middle pins wired to each other except the Arduino Tx-HC-05 Rx pins, which have a 1k/2.2k voltage divider circuit between then to shift the voltage down to ~3.3v.