IIRC Software Serial is not reliable above about 9600 baud, so 57600 is probably expecting too much.
Serial.print(nx[j], HEX);
When you print a value, it is cast to a 32-bit quantity. This will make it print C6 as FFFFFFC6 because C6 is a negative 8-bit quantity and the sign bit is replicated to the high order bit of the 32-bit word.
I think this will force it to print the hex properly:
Serial.print(nx[j]&0xff, HEX);
or it might also need this:
Serial.print((uint32_t)nx[j]&0xff), HEX);
I haven't got a working UNO/NANO any more so I can't test it (it works without the changes on a Teensy 3.2).
Pete