How can I make it stop at 255 and not just start from 1?
I assume the tricky part is happening in the header file, and it's really too advanced for me right now.
It would help if you actually posted the code or a link of where it is. The link you provided did not show what code you were using. The only code did not have the restrictions you obeserve.
How can I make it stop at 255 and not just start from 1?
Before you increment you test the value and only increment if the number is less than 255.
Be careful modifying library files, as you will cause them to be inconsistent with the released version, and then if you should update the library some day, all of your programs that rely on your modified version will break. It might be better to use the library function to read the value of the rotary encoder, then, within your program, check the value and compare it to a previous value and correct if it has rolled over.