I don't like to see long stretches of code within loop(). I find it much easier to follow and debug the logic if the code is put into different functions (which are called from loop()) so that each function just does one little piece of the project.
Have you tried testing the code without bothering with random numbers - perhaps pick numbers from an array so they are predictable and repeatable. Only switch to random numbers whene everything else works.
You have a number of looooonnnng delay()s that probably don't help.
How about just using the characters <>^v to show the user the directions?
...R