All right, so more learning and experiments later, I have found the problem that my board was experiencing, and I suspect this will help some folks in the future:
The issue that I was running into (and I suspect that @cclose above is experiencing) was that my GPIO8 was bridged to my GPIO9. Both pins affect the boot mode, and the documentation states this:
GPIO8 must also be driven High, in order to enter the serial bootloader reliably. The strapping combination of GPIO8 = 0 and GPIO9 = 0 is invalid and will trigger unexpected behavior.
In normal boot mode (GPIO9 high), GPIO8 is ignored.
Additionally, GPIO9 has a pullup on it. So if your IO8 and 9 are bridged. When IO9 is high (and 8 is also high), your board boots normally. If you pull IO9 down to boot into download mode, you are (unknowingly) pulling IO8 down as well. As noted above this results in unexpected behavior. In my case it was going into 'USB_BOOT'.
This problem is exacerbated by the fact that the pins are under the module, and you can't see if you have a bridge. I have test points on my board and that's how I discovered the problem. I used some flux and a hot-air rework station to reflow the solder. If you don't have a hot-air station, you might be able to put some flux on the edge of the board where IO8 and IO9 are, and put a soldering iron on, that might transfer enough heat to reflow it. I suspect this is an unfortunate side effect of module design coupled with at-home reflow practices (too much solder paste, unaligned stencil, imperfect reflow temperatures, etc.)
I hope this helps anyone else who runs into the problem.