The timing change suggests something different is happening at power-up-reset - the bootloader looks at the serial to check for an upload, potentially this could be affected by noise on that line, which in turn might be affected by capacitance, ground-plane, etc.
If you are using the 328p standalone it won't have anything connected to the RX pin (pin 0), so it will be floating and sensitive to the environment - try a 10k pull-up resistor from pin 0 to +5V maybe ?