Capture the point for children

So the main part that I am trying to work out is how to is this.

  • When you push the red or the blue button it takes 5 seconds to hold it and the same color LED will come ON. So you have two buttons red and blue and 2 RGB LEDs.
  • Now let's say the first captured color is red. The next child that comes goes to capture the blue light,
    He or she needs to hold the blue button for longer because he/she need to take the red light first(5-seconds), then after that there is a neutral time of 1-2 seconds where the light goes off then if he/or she holds it for longer than additional five seconds the blue light will come on(so in total 11 seconds needs to hold to go from red to neutral to blue)
  • The issue that I got is what kind of code to use when a child lets go of the button in the neutral or before the additional 5 seconds mark. The currently in the code it starts again from blinking blue to neutral to red
// using this to so that if the blue button was pressed after the red one it uses this IF for the red 
button, so that it takes you from blue to neutral to red
  if ((buttonR == HIGH) && (millis() - time > debounce))
  {
    if ((stateB == LOW) && (dur_on >= interval)) {

      digitalWrite(led2R, HIGH);
      digitalWrite(led2B, LOW);
      Serial.println("first led is on");
      state == HIGH;
    }

    if ((stateB == HIGH) && (neutral == LOW)) {
      if (dur_on <= interval) {
        digitalWrite(led2B, HIGH);
        delay(interval2);
        digitalWrite(led2B, LOW);
        Serial.println("losing blue led");
      }

      else if ((dur_on >= 5) && (dur_on <= 6)) {
        digitalWrite(led2R, LOW);
        digitalWrite(led2B, HIGH);
        digitalWrite(led2B, LOW);
        Serial.println("Neutral");
      }

      else if (dur_on >= interval4) {
        digitalWrite(led2R, HIGH);
        digitalWrite(led2B, LOW);
        Serial.println("red led on");
      }
    }
  }