That won't work. You can't add 0.1 to an integer and expect it to change. If pos is 0, pos + 0.1 is 0.1 which gets truncated to 0 when you store it in an integer variable.
The way the code is written the servo will start at around 90°. When you push a button it will go to 0° and sweep slowly to 180° or go to 180° and sweep slowly to 0°. Did you want the servo to stop where it is when you release the button or continue from where it last left off when you press the other button?