I need to deliver a constant 5V to some LEDs
Thank you very much for the help. For testing the circuit I skipped the current limiting resistors and was actually using a higher power LED I found lying around.
I tested the circuit now with the current limiting resistors and a lower power LED and it seems to be working as intended. Was the previous LED drawing too much current and somehow affecting the transistor performance?