Thanks guys, I pulled everything off the breadboard and redid my project, slowly more carefully and with shorter jumper wires, to avoid mistakes.
It works, so I think I just made a mistake somewhere earlier. I did buy a mosfet but now I have something higher current in mind.
But now the power draw when there is no motion is 48uA and when there is motion it jumps to 32mA.
Based on the 48uA and 2890mAH I think this should last 60,000 hours. Of course there is the whole self discharge thing I am not considering, so I will just replace the batteries once a year.