Do you have a pull-down resistor on the buttons? If not, then the inputs will be reading noise on the pins.
Are you sure it is counting in twos? I mean - if the button is bouncing or there is noise, there might be a '3' buried in there between the '2' and '4' but it's scrolling past to quickly for you to read it.
if(lastButtonCount != buttonCount) {
Serial.print(buttonCount);
lastButtonCount = buttonCount;
}