Button issue ATtiny84

I have i problem with getting a button to work on my programmed ATtiny84. All it is one button and an RGB LED every time you push the button and it go's to the next colour or pattern. The program works fine (tested with just the arduino everything working). The ATtiny is wired up the same as the arduino but isn't working the same it seems to be cycling throw the colour's its self with out me even pressing the button. Any idea's on why its doing this or how to fix it?