Okay, I tried doing the 10k between gate and 5V and same result. LED turns on without arduino power.
Interesting find, reversing the D and S gives proper performance when arduino is powered. Without arduino power it remains on.
Updated diagram
