Interrupts will run during delay, and do not disrupt the timing (unless they happen to run right at the end of the delay). One of the interrupts in question is the one that keeps track of milliseconds 
There is no reason to "lighten the load" on the microcontroller like that, it doesn't actually change anything as far as the microcontroller is concerned. Delay doesn't stop the micro or anything. I've seen that often done, and I think it's usually done because people put I/O related tasks into loop while their loop runs faster than the I/O device in question (for example, consider a sketch that printed one character to serial per run through loop, and the serial was running at 9600 baud. It takes the Arduino 1 millisecond to transmit that character, but the loop will run much faster than that, until the buffer fills up, at which point the serial.print() commands block until there's room in the buffer), or where you're relying on the loop only running X times per second. Both of these use cases can be more rigorously handled using the "blink without delay" type methods (search that if you aren't familiar with that sort of coding practice - it has been covered in excruciating detail countless times) without doing the delay.