That confusion is one major problem. Another is that you never read the button with a digitalRead(brightnessButton) A third is that pin4 is left floating.
Maybe look at the "How to Wire and Program a Button" stuff from here:
And for counting, try to understand the State Change Detection example from here:
The state change detection protects from repeatedly triggering the if() statement for as long as the button is active, instead, acting only when the button changes from being pressed to not-being-pressed & vice versa.
Work through the examples and learn how to understand what the code is doing, and then maybe you can work through your code and make sure it does what you want.