So you just forgot to mention that you had the regulator removed.
That makes part of my answer invalid.
Your setup can never work the way you have drawn it.
With removed regulator, the RAW pin is not attached to anything anymore.
Unless you forgot to mention something else.
If you are going to power both MCU and servo from a single set of batteries, then use good power supply decoupling.
I suggest 1000uF or more.
Leo..