This part here is inside the IF statement and will only set the position back to 180 if your holding the button.
if ((pos == 0) || (pos < 0))
{
pos = 180;
delay(5); // control speed to restart position
}
What you need is to have a ELSE statement that tells the code what to do when you are not pressing the button or if you press then release the button.
Look into State machines.