ATMEGA32U4 5V regulated and USB powered PCB layout

Ok Im working on a project where i use the Atmega32U4 and i have it pretty well laid out on the PCB already. I have a Regulator IC that allows me to use 4 AA batteries to power this project. The IC gives me regulated 5V output and works good however I want the option of powering the project over USB.

The USB side is laid out other then the Vbus line. Do i need to use a diode or come up with a circuit using mosfets so that when USB is plugged in theirs no kind of problems. I thought about putting Vbus in parallel with the 5V line but im worried there may be some issues.

If anyone can help at all please let me know.

Rob