Because e.g. \x86aa is a valid integer and \x86ii is not ? So the compiler treats it like that and next truncates to put it in an unsigned char.
These are the warnings that I get:
C:\Users\sterretje\AppData\Local\Temp\arduino_modified_sketch_52481\sketch_may06a.ino:9:3: warning: hex escape sequence out of range
"\x87\x86aa",
^
C:\Users\sterretje\AppData\Local\Temp\arduino_modified_sketch_52481\sketch_may06a.ino:10:3: warning: hex escape sequence out of range
"\x87\x86bb",
^