Arduino auto start problem

Is your Button on a long wire ? that can also act as an aerial for interference.

If you do not have a schematic, draw it one a piece of paper and photo it, also a photo of the actual hardware may help as the physical layout can be important.