I see in your code you have:
if (analogRead(AnalogKeyPin) < 850 ) // if a keypress is present
which begs the question: Why use an analog input for a digital button press??
Also, you need to take some time to learn how to properly format and indent your code. Your code is rather difficult to follow (for me, at least).
As for your question, try this instead:
if (xVal > (center + threshold))
return RIGHT_KEY;
else if (xVal < (center - threshold))
return LEFT_KEY;
if (yVal > (center + threshold))
return UP_KEY;
else if (yVal < (center - threshold))
return DOWN_KEY;
if (buttonstate == HIGH)
return SELECT_KEY;
return NO_KEY;