I recommend connecting AVCC to power and ground the GND pin on that side of the IC. I don’t know if it is concerning your bootloading gone wrong, but couldn’t hurt to try. As I understand it without decoupling caps the MCU will not be stable. Where did you get the bootloader hex file and is there any documentation that came with it such as what are the fuse requirements and what pin if any does it blink?
One more thought and then I’ll be quiet, there are some interesting tips in this post although the OP didn’t write back to say what fixed it, if anything. https://forum.arduino.cc/t/atmega1284p-flashes-bootloader-but-nothing-else/1239124