When designing a customised board with an AVR on it (amega328p, attiny85...) what is the smallest allowable value for the pullup from the reset pin to the 5V Vcc rail? I know the pin has an internal pullup of somewhere from 30K to 100K ohms, and 10K is pretty common for an external pullup. But if you want a custom board to still work whilst close to electrically noisy things (motors for example, or sitting very close to a switch mode power supply), how small a pullup can be used whilst still being a large enough resistor to enable an ArduinoAsISP programmer to be able to bootload and flash programs to the board?
I do not think that there is one answer to that question. I would say that the pullup would depend on the electrical noise in the system. So a case by case solution would be necessary.
My opinion is that if you have to go lower than about 5K its time to try to mitigate the noise, not use a stronger pullup.
Makes sense for an answer, but is there an absolute limit from looking at it the "other way round". How small a pullup would make programming impossible? You say 5K is still large enough for programming to be done, and I've done a testwhere I seem to be able toprogram with a 1K pullup present (though I can't be sure if this is reliable or just good luck). How low can one go? From the perspective of an ArduinoAsISP what is it doing with the "pin 10" it uses to control the reset of the chip being programmed, would current limits on this arduino's pins be the best thing to inform what the minimum possible pullup would be?
But again, I would not go lower than 5K unless there were no other way. If you have that much noise, it's not just the programming or holding the processor out of reset that would be affected, the normal operation of the processor would probably be affected and cause havoc or hard to diagnose problems. Fix the source of the noise. Use separate power supplies for noisy things like motors and other inductive loads, optical isolation of signals, additional bypass and bulk capacitance, shielded cabling, proper wire rounting and so on.
When designing custom boards with any microcontroller, it's important to thoroughly study the manufacturer's datasheets, application notes, etc - they will cover this stuff.
RESET pin can act as normal I/O which is limited to 40mA current according the datasheet as absolute maximum. However, in the reset functionality, which is default BTW, it can be connected directly to the VCC so 0Ohm pull-up resistor literally. In this case, things like the reset switch do not have much sense, right?
Anyway, this is probably not very helpful to your problem with "electrically noisy things". I would just follow what was already written by @groundFungus and @awneil .