String myString = "";
myString = String((char *)payload);
Serial.println(myString);
const char* hexstring = myString.c_str();
uint16_t hex = atoi((char *)hexstring);
This is uselessly pissing away resources. The payload array is the same size as a char array. All that it is missing is a NULL terminator. Wrapping it in a String adds a NULL terminator, but who knows where.
payload[length] = '\0';
Puts it in the proper place.
Then, you can just pass payload (cast to char *) to atoi:
uint16_t num = atoi((char *)payload);
However, that won't work, because atoi() expects the string to be a base 10 representation of an int. strtoul() can handle a string in any base.