Hint: Schematics follow normal reading patterns, top to bottom, left to right, in on the left out on the right. Positive top, negative bottom. This puts the signal flow basically left to right.
Your schematic is ok but a little more difficult to follow. @LarryD describes the best way to draw the schematic, much less prone to errors. @sonofcy gives you a good solution.
Why don't you use an ESP32 with battery connector and buil-in charger.
The XIAO ESP32-C3 is such an example.
Solder the battery wires on and plug in USB, that's it.
Leo..