Can't upload a sketch without bootloader for second time, just on

Hello,

I want to programme an ATmega32u4 by ISP with an Arduino Uno board. The micros that I used were new, and I have programmed two ones with this method. The issue is when I try to programme the second or third time, I get an error:

Device signature = 0x000000
avrdude: Yikes! Invalid device signature.
Double check connections and try again, or use -F to override
this check.

Could you please help me with this?

Note: I am using a custom pcb, in which I put the Atmega32u4 (smd) and push with my finger at the same time that I upload the sketch.

If you program via ISP it over writes the bootloader.