Why not just get a switching 5V regulator? Then you won't see any variation.
http://www.dipmicro.com/store/DCA-0510
http://www.dipmicro.com/store/DCA-0520
Or get 6V, 7.5V if you want to regulate it down yourself. Maybe use an LM317 with a trimming resistor (10 turn potentiometer) to get it right at 5v.