motorSpeed = analogRead(potPin)/4;
Why did you do this ? Have you heard of the Map function ?
Have you measured the motor voltage with a meter while running your code ?
Write a simple test sketch that just drives the motor at a speed set by the pot using the Map function and nothing else.
No switches.
No nothing. Just a direction command and a speed command using the Map function.
Take a photo of YOUR circuit and post it . Make sure we can see all the wire connections.