analogWrite not functioning in if statement

... it's basically resetting brightness every time it goes through the loop ...

void loop(){
  int brightness = 0;

Yes it is. You could move it outside:

int brightness = 0;
void loop(){

Now it only does it once.