guarda-chuva:
I understand the concept, but I don't know the benefits and downsides
Benefit: Less external components you need to wire up for you switches. Also the thing in Perry's post.
Downsides: None.
I feel like the first one is more "correct", but I'm not sure. Which one is better to use?
It's exactly as correct as always driving on the right-hand side of the road. 
Which is to say, it is not something that is objectively correct, but merely an arbitrary convention that you are establishing in your head. Similarly, there is nothing about a particular voltage level that inherently associates it with ON or OFF. In fact, a massively huge number of chips use the "active low" convention for their inputs and outputs.
Sometimes different circuits will use different conventions. One project I made is a persistence of vision display that uses an active low Hall sensor (output goes LOW when the magnet is close) and LEDs wired to be active high (turn on when the output is HIGH). Just like you define names for pin numbers, you can define names for levels and other things you need help keeping track of, like this:
const uint8_t HallEffectPin = 10;
const uint8_t HallActiveLevel = LOW;
const uint8_t LEDPin[] = {2, 3, 4, 5, 6, 7, 8};
const uint8_t LEDPins = sizeof(LEDPin) / sizeof(LEDPin[0]);
const uint8_t LEDActiveLevel = HIGH;
So when I want to check if my Hall sensor has a magnet near it, digitalRead(HallEffectPin)==HallActiveLevel is much more informative to read than digitalRead(HallEffectPin)==LOW. And if I want to turn off an LED, I don't have to look up which way they're connected, I can just do digitalWrite(LEDPin[pin], !LEDActiveLevel). With this coding practice, you can easily use either convention in your code, and even use multiple conventions for different attached circuits when necessary.
lesept:
To me, active LOW seems natural, because I push the button. So pull-up makes sense.
This is not the right way to think about it either. It is best to decouple the two concepts in your mind so that you are comfortable using either convention when necessary.
