The "Solution" didn't work. I got a bit overexcited and should have tested more than once before posting that. I think it so happened that the commutator was in a favourable startup position or something, because twice now it has worked by shear luck. I'm on my lunch break now but when I finish work I'll try a bigger cap - I don't think that's the proper fix though.
I woke up in the middle of the night and thought why not just use the TP4056 (actually the DW01 battery protection chip) to run the base of a large PNP. That way I can pull whatever current I desire within reason and the battery undervolt protection will still work.
I'll let you know how I get on. I've been googling like made, but can't find anybody else describing the same problem.