The flush() method does not clear the receive buffer.
For the problem that have, what cirrently happens is that you control the outputs briefly based on serial and next briefly based on the in the button.
Use a flag variable (e.g. a bool) and set that to false in the beginning of loop. Set it to true if the correct serial data is received or when the button is pressed.
Move the controlling of the outputs to a separate if block after the the reading of the serial port and the reading of the button.