You don't need 12V, much of the power is just wasted as heat by the 5V regulator.
AA battery is usually 1.5V, not 1.2V. Are you using some kind of rechargeable that only puts out 1.2V?
5 x 1.5V = 7.5V, use that into the regulator, it will run much cooler.
Or 6 x 1.2V.
The regulator will only supply 800mA maximum before it overheats and shuts down. With 12V in, it will overheat sooner. With 7.2V to 7.5V, you have a chance for getting closer to 800mA.
How much current do your sensors and motor drivers need?
When they are not connected, do you see 5V output?