When something is going wrong, the key to troubleshooting is to eliminate as many unnecessary variables as possible. With the sketch you're using, the problem could be a bug in the code, it could be that the buttons are not correctly wired, it could be that the servo is not correctly wired.
So you need to test each component individually. Start with a simple sketch that just moves the servo back and forth. You can find one written for you at File > Examples > Servo > Sweep. If the servo moves with this sketch uploaded, then you know the servo is wired correctly and is working.
Next would be a simple sketch to test your buttons. Generally, I would recommend a sketch that reads the button pin and prints the state to the Serial Monitor. But I don't think you have a serial to USB adapter connected to your ATmega328P, so you don't have that output option. Next best would be an LED that turns on or off depending on the button pin reading. If you don't have an LED, then you'll have to use the servo as an output to indicate whether the button is being read correctly.
One common mistake people make when starting to work with bare microcontroller chips is thinking the physical pin numbers are the Arduino pin numbers. They are not. You need to study the pin mapping to know which physical pins on the microcontroller are mapped to each Arduino pin number:
You can see that, for example, Arduino pin 4 is physical pin 6 on the chip (PD4).