It is right in the opening post. User gcjr made the kind suggestion that the reading of the eprom over and over again in the loop is pointless so I removed it there and added the value that is being put in the eprom after the relays have switched in the flag variable.
Same result, but less reading of the eprom in the loop.
I tried using HIGH and LOW for this part before, but it doesn't work in combination with the Eeprom. the digitalRead is reading a variable, not a pin.