Multiplexing LED only works with serial.print() but not with delay()

Using Arduino LEDs and Multiplexing
6

Okay, so it seems changing

if (! rtc.begin()) {
  //Serial.println("Couldn't find RTC");
  while (1);

to

rtc.begin()

and removing Serial.begin as well as all Serial.print lines resolved the issue. I still can't really wrap my head around it though. I'm not using Pin 0 or 1.