But isn't that what the next 2 IF statements do? Test in the 0-4 seconds for the max value and test in 4-8 seconds for the min value?
The millis() function returns a relative time. The time is relative to when the Arduino was reset. You are using it as though it was some sort of absolute time.
Your watch is another relative device. It tells you what time it is relative to some arbitrary event that occurred long ago. Yet, you'd have no problem showing up if I said "Meet me at the bar in 20 minutes, and I'll buy you a drink." You'd have no problem answering if we were supposed to meet there at 2:00, and I was late, and I asked you how long you'd been waiting.
You need to use millis() the same way. Set a variable with the time that calibration started. Test whether now (millis()) minus then (the stored time) is greater than, or less than, some interval.