Do you have a deadband in the middle for all motors off? If all motors are off when the joystick is in the deadband, then how would you want "down" to be interpreted by the motors? You can't get any slower than off.
This (above) is good. Swap pin-pairs in the code or hardware if you want fans to reflect stick movement.
// If joystick stays in middle the motors are not moving
else {
motorSpeedA = 0;
motorSpeedB = 0;
}
This says "if your Y hits the deadband, stop the motors... even if you push up or down"
What if you use the SEL (joystick pushbutton) as "motors on/motors off", the deadband zone is for a moderate speed (both motors), UP is increase PWM, DOWN is decrease PWM, LEFT is left motor and RIGHT is right motor.
I suspect your wiring.