Trying to use button holds to change states

Here is the code, with some extra stuff because I was just trying to experiment with buttons:

/* This is an example sketch provided to test the application of dual button commands
 as well as the application of a requirement for the user to hold down certain button(s)
 for a specific amount of time. */

int programmingHold = 2000;
unsigned long startHoldTime;
unsigned long time = millis();
int buttonOne = 11;
int buttonTwo = 12;

void setup()
{
  Serial.begin(9600);
  pinMode(buttonOne, INPUT);
  pinMode(buttonTwo, INPUT);
  pinMode(13, OUTPUT);
  pinMode(2, OUTPUT);
  digitalWrite(2, HIGH);
  pinMode(3, OUTPUT);
  digitalWrite(3, HIGH);
}

void loop()
{
  int button = digitalRead(buttonOne);
  Serial.print(button);
  Serial.print(" | ");
  Serial.print(time);
  Serial.print(" | ");
  Serial.print(startHoldTime);
  Serial.print(" | ");
  Serial.print(time - startHoldTime);
  Serial.println();
  delay(2);

  if (button)
  {
    startHoldTime = time;

    if (button && ((time - startHoldTime) >= programmingHold))
    {
      digitalWrite(13, HIGH);
    }
  }
  else if (button == LOW)
  {
    startHoldTime = time;

    if ((button == LOW) && ((time - startHoldTime) >= programmingHold))
    {
      digitalWrite(13, LOW);
    }
  }
}

The idea being that if you connect power (in this case I'm just using a jumper to connect pin 2 to pin 11, but the idea would be to use a button) for 2 seconds, it will turn the LED on pin 13 on, and if you remove power for more than 2 seconds, the LED will then shut off. The code is not working at all, currently, but even when I remove the timing aspects, as seen below, my serial monitor shows that pin 11 is reading as HIGH for several seconds after I remove power.

/* This is an example sketch provided to test the application of dual button commands
 as well as the application of a requirement for the user to hold down certain button(s)
 for a specific amount of time. */

int programmingHold = 2000;
unsigned long startHoldTime;
unsigned long time = millis();
int buttonOne = 11;
int buttonTwo = 12;

void setup()
{
  Serial.begin(9600);
  pinMode(buttonOne, INPUT);
  pinMode(buttonTwo, INPUT);
  pinMode(13, OUTPUT);
  pinMode(2, OUTPUT);
  digitalWrite(2, HIGH);
  pinMode(3, OUTPUT);
  digitalWrite(3, HIGH);
}

void loop()
{
  int button = digitalRead(buttonOne);
  Serial.print(button);
  Serial.print(" | ");
  Serial.print(time);
  Serial.print(" | ");
  Serial.print(startHoldTime);
  Serial.print(" | ");
  Serial.print(time - startHoldTime);
  Serial.println();
  delay(2);

  if (button == HIGH)
  {
    startHoldTime = time;
    digitalWrite(13, HIGH);

  }
  else if (button == LOW)
  {
    startHoldTime = time;
    digitalWrite(13, LOW);
  }
}

I'm very confused as to why the pin seems to still be getting power several seconds after I remove all connection from it. It has to be something I'm missing, but I'm just having a hard time understanding what is going on.

Hassurunous:
Here is the code, with some extra stuff because I was just trying to experiment with buttons:

I don't think there is any point in trying to capture time = millis() BEFORE setup()
I suggest you move that line to the bottom of setup().

But I doubt if that is the real problem.

You need a variable to record whether the led is on (say) ledState = 0

this pseudo code explains the process

IF ledState == 0
       You need to capture time when the button is  pressed AND when it is released
        AND you need to check whether your 2 secs has expired
               it will reset if the button is released
         IF the button is down and the time expires
          CHANGE the variable to ledState = 1

almost the same process is required to turn the led off

IF ledState == 1
       You need to capture time when the button is  pressed AND when it is released
            use a different variable for the time so it can't be mixed up with the other part
        AND you need to check whether your 2 secs has expired
               it will reset if the button is pressed
         IF the button is UP and the time expires
          CHANGE the variable to ledState = 0

and after all that you just have

digitalWrite(ledPin, ledState);

The usual way to manage the time is like this

when button changes
ledOnMillis = millis()
and to see if the time has passed
if (millis() - ledOnMills >= ledOnWaitInterval) {

...R

Why not just toggle the LED? That way the previous state is irrelevant.

Using Direct Port manipulation:

LED, pin 13 (Uno)

void setup(){
pinMode(13, OUTPUT);
}

void loop(){
//This function toggles pin 13
PORTB = PORTB ^ (1<<PORTB5);  // toggle bit 5, leave others untouched
delay(500);
}

Pin mapping:

Why are you using pin 2 to supply pin 11 with 5V when the 5V output will do the same and leave pin 2 free for other things?