It depends on which Arduino you have. Many of them have a buffer on pin 13 to drive the LED. If the LED is directly connected to the pin and you need to use 13 as an input, the inputting device is forced to drive the LED.
If you have a soldering iron you can attempt to remove the LED. You will always destroy it in the process, so there's no need to be gentle. On many Arduinos, such as the Micro, it's actually easier to remove the resistor instead of the LED. Even without a soldering iron you can grip it with pliers and tear it off the board, but that has a risk of damaging other parts nearby.