Basic maths: 10k? resistor, 5V voltage, 100µF cap=0.76 second LED fadeout

How come you're not using a transistor ? You can make it fade out nicely then!