So the value of this resistor needs to be high enough in order to not drain too much current when the button is pressed
but in the same time should not exceed a certain value so as to the input voltage on the pin is high enough.
No. The voltage is either ~5V or ~0V (ground potential) no matter what resistor value is chosen.
If the resistance is too high the input will be vulnerable to noise pick-up. There are electrical signals all around. The strongest is usually from power lines in your house/building. For example, if you touch the signal input to an audio amplifier, you'll get a 50Hz or 60Hz hum/buzz. That's your body acting as an antenna and picking-up the power line "signal".
There was a post recently where someone was getting an interrupt triggered 50 times per second... Power line noise into a high impedance input!
There is a limited amount of energy in those signals. Since higher impedance/resistance requires less current, you'll get higher noise voltage with higher resistance. And, it's the noise voltage that can give false/wrong input. The noise current mostly flows through the pull-up/pull-down resistor, since it has lower resistance.
This is why as a compromise people use a resistance whose value is 1/10 of the MCU internal impedance .
The resistor has be low enough to "overcome" or dominate the internal impedance.... The pull-up resistor and the internal impedance form a [u]voltage divider[/u] and if the pull-up is much lower than the internal impedance (almost) no voltage will be "lost" across the pull-up and it will pull fully-up or full-down (and you can ignore the voltage divider effect).
Although, without studying the chip you don't know if that internal impedance provides a path to ground, or a path to Vcc, or maybe both... So in some cases (such as with the Arduino) you don't strictly have a voltage divider.
The Arduino has an input impedance of around 100 Megohms, (which we can consider "almost infinite" impedance for practical/engineering purposes). 1/10th of that would be too sensitive to noise pick-up.
10K is usually a good compromise. The pull-ups built-into the ATmega chip (which can be optionally enabled) are between 20K and 50K.
When the button is released, I don't see in what this resistance is usefull. Without it the pin of the MCU would still be at the low state
No. The Arduino input is "floating" and undefined when open. I think they tend to float high, but I'm not sure and the datasheet just says undefined.
It depends on the particular part/circuit. TTL logic devices are high when open/unconnected (although it's considered good engineering practice to add a pull-up, and/or to ground unused pins).
Another issue is that there is some input capacitance. A high pull-up or pull-down resistance along with the capacitance will form a low-pass filter and it won't pull-up or down as fast when the switch is open. (With a manual switch, a few microseconds of delay won't be noticed.)
As a final question would it be right to imagine a pull-down resistor made like this :
Not quite... The pull-up is always connected to the input. You've got it on the wrong side of the switch.