delay() is pretty accurate, but fairly useless for a variety of other reasons. One of them of course being the problem you are running into. It's rather useless when used in conjunction with the rest of the program to perform overall timing.
Take a look at the Blink without Delay sample. That provides a good method for overall timing.