No delay value smoother

    if(fade <= 0)
    {
      fadeIncrement +=5;
    }
    if(fade >= 255)
    {
      fadeIncrement -=5;
    }

    Serial.println(fade);

So, if fade is less than or equal 0, change fadeIncrement. If fade is greater than or equal 255, change fade increment. Then, print fade. Since fade started at 0, of course it will print 0 the first time.

  fade = fadeIncrement;
  analogWrite(ledPin, fade);

If you increment, instead of set fade, fade will gradually take on different values.

Of course, that should happen INSIDE the "if it's time" block. And, before the print of fade.

  unsigned long time = millis();
  potInterval = analogRead(potPin);    

  if(time - preTime > potInterval)

Picky, I know, but I like names like currTime and prevTime so it is clear which is which.

Oh, and setting prevTime to currTime somewhere would be useful, too.