Time to write to a pin blows up when setting infrequently

I am trying to drive speakers at certain frequencies. I know that it has been done many times in the past, but I want to figure out how to do it myself.
However, I am running into an odd problem and I wrote a small test to showcase it.

unsigned long behaviorTest(long iters, int prop){
  unsigned long startTime = micros();

  for(long i=0; i<iters; i++){
    if(i%1 == 0){ // Modulo is to make the pin write more or less frequent
      PORTD ^= B00000001 << 3; // Manually toggles one pin
    }
  }

  unsigned long endTime = micros();
  return (endTime-startTime)/(iters/1000); // Returns the average number of nanoseconds it took to execute one loop
}

I then vary the if statement modulo in order to change the number of times the pin will be toggled in the test.

Logically, I would assume that the more times the pin is toggled, the longer it would take to execute the function. However, I observe the opposite effect in practice.

When the pin is toggled every loop, the average loop time is 560 nanoseconds.
When it toggles every other time, it is 1006 nanoseconds.
Every 5th time, it is 39946 nanoseconds. It remains about constant at this number until it is running once every 100000 time, and then it drops back down.

I am completely stumped on what could be causing this. Why does performing a bitwise operation on PORTD take so long? I assumed it would be in constant time like setting any other variable. I feel like a toggle should take less than 40000 nanoseconds, but I'm no expert and can in no way explain this behavior.

Did you take into account that (i%1) is always zero, and that (iters/1000) is zero for iters<1000?

Hint: a quick way to toggle pin PORTD.3 on an AVR-based Arduino is:

PIND = 1<<3;

The compiler probably optimizes %1 and %2, while other numbers result in the actual division routine being called (which take much longer than toggling the pin)