That code layout is hideous - the only way I can see which braces pair up using using an editor that matches them for me. And I'd much rather you used boolean values true and false for your boolean variables, instead of using the values defined by Arduino for digital I/O states. Having said that, I don't think either of these issues are causing your problem and look at length mode as an example the only place that length is updated is an assignment from encoder0Pos, and encoder0Pos is set from length at the start of length mode. The code looks pretty complex (and I can't help thinking some of that complexity is not required) so I can't be sure that encoder0Pos is updated correctly, but nothing jumps out at me as likely to cause the length variable to be set to 4.
I think MarkT's thought about the board resetting is the most likely explanation and is what I'd check for first. You could explicitly check for a reset by making the board do something obviously recognisable during setup(). Another way to check is to change the initialisation of length to some other arbitrary value. If the symptoms change and it now jumps back to that new value, it's pretty strong evidence that the board is resetting.