4 Digit 7 Segment Display randomly turning on and off for no reason

I just wrote a presumably simple code trying to figure out how to use 7 segment displays but it's not working as I wanted.

I am controlling the display via a 74HC595 shift register using pins 8-10 and turning the 4 single digits on and off using pins 2-5.

The display is working fine but it turns off every couple of seconds and then turns back on after 1-3 seconds. This does not always happen at the exact same times. Sometimes it starts after 15 seconds after resetting, sometimes after only 5 or even 2 seconds.
In the CountUp function I check if one second has passed and increment the variable for the first digit. When it reaches ten it gets reset to 0, the variable for the second digit gets incremented and so on for all 4 digits. Should be simple. Should. But it seems this function is causing the problem.

I have tried commenting out calling CountUp() in the loop and the problem no longer appeared.
So it's probably something with the CountUp function. Calling it every tick (frame? How do you call that in electronics?) shouldn't be too heavy on the performance either since it's basically just comparing two numbers. The var lastmillis should be large enough too since it's an unregistered long.
I really don't get what's the problem here.

Edit: It is NOT the CountUp function. I tested leaving calling CountUp() in the loop and changing the Display to just pass through some numbers as parameters instead of the variables.

void loop() {
  CountUp();
  Display(2,2,7,7);
  //Display(s1, s10, s100, s1000);

}

And it's just getting weirder. It seems like passing through certain numbers causes problems but others are ok. 1,1,1,1 works just fine but 1,1,1,2 causes the bug. 9,8,7,7 works but 9,8,7,6 is buggy. I don't get this at all.

int latch=9;  //74HC595  pin 12 STCP
int clock=10; //74HC595  pin 11 SHCP
int data=8;   //74HC595  pin 14 DS

int dig1 = 2;
int dig2 = 3;
int dig3 = 4;
int dig4 = 5;

int s1 = 0;
int s10 = 0;
int s100 = 0;
int s1000 = 0;
unsigned long lastmillis= 0;



unsigned char table[]=
{B00111111,B00000110,B01011011,B01001111,B01100110,B01101101,B01111101, B00000111,B01111111,B01101111}; //0,1,2,3,4,5,6,7,8,9

void setup() {
  pinMode(latch,OUTPUT);
  pinMode(clock,OUTPUT);
  pinMode(data,OUTPUT);

  pinMode(dig1, OUTPUT);
  pinMode(dig2, OUTPUT);
  pinMode(dig3, OUTPUT);
  pinMode(dig4, OUTPUT);
}
void Display(int ps1,int ps10,int ps100,int ps1000)
{

  digitalWrite(dig1, HIGH);
  digitalWrite(dig2, HIGH);
  digitalWrite(dig3, HIGH);
  digitalWrite(dig4, LOW);
  digitalWrite(latch,LOW);
  shiftOut(data,clock,MSBFIRST,table[ps1]);
  digitalWrite(latch,HIGH);
  delay(2);

  digitalWrite(dig1, HIGH);
  digitalWrite(dig2, HIGH);
  digitalWrite(dig3, LOW);
  digitalWrite(dig4, HIGH);
  digitalWrite(latch,LOW);
  shiftOut(data,clock,MSBFIRST,table[ps10]);
  digitalWrite(latch,HIGH);
  delay(2);

  digitalWrite(dig1, HIGH);
  digitalWrite(dig2, LOW);
  digitalWrite(dig3, HIGH);
  digitalWrite(dig4, HIGH);
  digitalWrite(latch,LOW);
  shiftOut(data,clock,MSBFIRST,table[ps100]);
  digitalWrite(latch,HIGH);
  delay(2);

  digitalWrite(dig1, LOW);
  digitalWrite(dig2, HIGH);
  digitalWrite(dig3, HIGH);
  digitalWrite(dig4, HIGH);
  digitalWrite(latch,LOW);
  shiftOut(data,clock,MSBFIRST,table[ps1000]);
  digitalWrite(latch,HIGH);
  
}

void CountUp()
{

if (millis() - lastmillis > 1000)
{
 lastmillis += 1000; 
 s1++;
 if(s1>9)
  {
    s1=0;
    s10++;
    if(s10>9)
    {
     s10=0;
     s100++;
      if(s100>9)
      {
        s100=0;
        s1000++;
        if(s1000>9)
        {
         s1000=0;
        }
      }
    }
  }
}


  
}
void loop() {
  CountUp();
  Display(s1, s10, s100, s1000);

}

Delay could be causing the issue. If too many delay are run concurrently, it can cause the display to stop working for moments.

But there just miliseconds and it turns off for a second or two.
And how would I do it without a delay? If I didn't have delays between turning on each digit it would only turn on the last one, right?

And as I said, the bug only appears with certain combinations of numbers. For example, Display(2,2,2,2) works fine but with Display(2,2,2,7) it starts turning on and onff again. This really makes no sense to me.