I would say it's better to have the diode than not have the diode, and for the cost of a diode, it's not worth worrying about.
Yes, the voltage will droop as the current draw exceeds the supply, but the input voltage will droop more and sooner than the output voltage, so you can set your switching level to be that more reliable voltage droop. I would also think about adding some kind of delay to the switching, or it might oscillate:
Light drops
Current demand exceeds supply
Voltage drops
Switches to wall wart
Current demand drops to 0
Voltage Rises
Switches to solar panels
Current demand exceeds supply
Voltage drops
Switches to wall wart
Current demand drops to 0
Voltage Rises
Switches to solar panels
... etc