menu case problem with starting up

      Serial.print("ENTER");

This should be println(), to add a carriage return/line feed.

int loopConstrain(int x,int a,int b){
if (x == a-1 || x ==255){
   return b;
 }
 if (x == b+1){
   return a;
   }
   Serial.print("x=");
  Serial.println(x);
  Serial.print("a=");
  Serial.println(a);
  Serial.print("b=");
  Serial.println(b);   
   
}

Most compilers would output a warning that says that not all paths return a value. If there no constraining taking place, there is nothing returned from this function. If there is constraining, nothing is printed.

You should endeavor to write functions that have exactly one return statement, and that should be the last statement in the function, like this:

int loopConstrain(int x,int a,int b)
{
   int constrainedValue = x;
   if (x == a-1 || x ==255)
   {
      constrainedValue = b;
   }
   else if (x == b+1)
   {
      constrainedValue = a;
   }
   Serial.print("x=");
   Serial.println(x);
   Serial.print("a=");
   Serial.println(a);
   Serial.print("b=");
   Serial.println(b);
   Serial.print("constrainedValue=");
   Serial.println(constrainedValue);
   return constrainedValue;   
}

The lack of the return statement in most instances of the call to your function meant that garbage was popped off the stack as the return value.