Automating String Pinsetter for Half Scale Bowling Alley

Good job getting the compiling issues cleared up.

Yes, Your use use of millis() for timing is fundamentally wrong in the section you highlight, and is different from the other uses of millis() for timing in the code.

In Mode 1

if (ballRolledState == !lastBallRolledState) { // if state changes to LOW, ball has been rolled
        ballRolledTime = millis(); // sets time ball was rolled
        Mode = 2;
      }

But then in Mode 2 you have this.

int time = ballRolledTime - millis(); // Keeps track

As millis() grows larger and larger, the value of time will get more and more negative, yet your conditionals in this section depend on a value greater than something.

A careful review of this millis() tutorial will be helpful
https://forum.arduino.cc/t/using-millis-for-timing-a-beginners-guide/483573