You have way too much code when you are not even sure that all the hardware works. Dump ALL of the code that deals with the joystick, INCLUDING the library. Test JUST the switches. Print something that confirms that a switch is pressed or is not pressed. When that consistently prints the correct value, after the lag clears up, implement the state change detection code. Then, add the joystick stuff back in.
Post a wiring diagram - hand-drawn is fine.