serial.print junk while counting button presses

Do you have a pull-down resistor on the buttons? If not, then the inputs will be reading noise on the pins.

Are you sure it is counting in twos? I mean - if the button is bouncing or there is noise, there might be a '3' buried in there between the '2' and '4' but it's scrolling past to quickly for you to read it.

if(lastButtonCount != buttonCount) {
  Serial.print(buttonCount);
  lastButtonCount = buttonCount;
}