The problem is pressing the down key and holding, will it run the case for the UP key.
You need three cases, obviously, AND you need to keep track of the last key pressed (the one before the repeat code appears) so you know what is being repeated.
delay(500);
Not a good idea...