I'm guessing the 5V source may be battery? The 12V from the car? It's not a good idea... you need a way to switch it... a comparator circuit that senses 12V at Vin should work.
I've done something simlilar before and it's doesn't work well and can drain the 5V source and if that's battery it's not a good thing.