Thanks for your reply!
I just tried it, but unfortunately it makes no difference. I use the serial.begin so I can see if something is wrong with the setup of the HC-05 module. I deleted all lines in the code which used the serial interface including Serial.begin in the beginning. The program still works as it should but the LEDs are still on.
As for unsoldering the LEDs: While it might be an option, I find the LEDs for the bluetooth communication quite helpful, so I might just leave them on. The other two on the other hand do not have any benefit for me in this application so I might just unsolder these. Unless this causes any other problems with the board, I'm unaware of?
Still if anyone has a solution on how to get by with this problem without unsoldering the LEDs I would prefer that. Just for re-using the Arduino after I am finished with this project.