With this code the LED only blinks when Serial Monitor is on and stays on while Serial Monitor is off. Another problem I have is that if I comment out the current digitalWrite(LED_BUILTIN, HIGH) and replace it with the one I have commented out then the LED wont blink even if Serial Monitor is off.
The micro uses ATmega32u4 which has a native USB interface. After Serial.begin() you must wait until the USB connection is established to get it working:
void setup() {
Serial.begin(9600);
while( !Serial ); // wait until USB connection is running
pinMode(13, OUTPUT);
If you don't have a USB connection you need a timout to detect that and should not use Serial.print in that case.
Edit: I think what you see blinking when the monitor is open ist the transmit led, not the led on pin 13. Without the second delay, this led will never blink ( at least you cannot see it ).