Detecting special characters in string

Projects Programming
1

I'm working on a project using an ESP8266 in combination with MAX7219 and the MD_PAROLA library to display information. Things are working properly, with one notable exception: special characters.

Specifically, I can't get the degree symbol to render properly. I understand that this is related to unicode, and that I need to pass in the hex value rather than the character, but I'm not sure how to go about detecting the presence of the degree symbol in the string received, and modifying the character array sent to the display to render it properly.

When outputting the character array newMessage to the serial monitor, it renders properly, but on the display it does not.

Any help is greatly appreciated!

bool processJson(char* message) {
  StaticJsonBuffer<BUFFER_SIZE> jsonBuffer;

  JsonObject& root = jsonBuffer.parseObject(message);

  if (!root.success()) {
    PRINTS("parseObject() failed");
    return false;
  }

  if (root.containsKey("message")) {
    root["message"].as<String>().toCharArray(newMessage, BUF_SIZE); // This is not handling special characters properly
  }

  if (root.containsKey("repeat")) {
    PRINTS("---------------------- root.repeat ");
    repeatMessage = root["repeat"].as<int>();
    PRINTLN(repeatMessage);
  }

  if (root.containsKey("textEffect")) {
    P.setTextEffect(effects[root["textEffect"]["in"].as<int>()], effects[root["textEffect"]["out"].as<int>()]);
  } else {
    P.setTextEffect(PA_SCROLL_LEFT, PA_SCROLL_LEFT);
  }
  return true;
}
2 
    root["message"].as<String>().toCharArray(newMessage, BUF_SIZE); // This is not handling special characters properly

What part of that is not handling the special characters properly? You are invoking the [] operator, to get something. You pass that something to the as function, which does something we can only guess at. You call the toCharArray() method to convert the chars to an array of chars. That method does NOT handle characters that are not in the range -128 to 127.

What value does the character that is not "handled properly" have?

3

I'm struggling to even understand the value that the degree symbol has. It renders in the MD_MAX7219 as a character that appears to be a grid of dots. It renders in the serial output as a degree symbol.

The ASCII value of the degree symbol is well above 127 (according to Wikipedia, it's 248). I understand that this means the character actually takes more than a single byte to encode.

The issue is that I'm unclear on how to detect the first byte of this special character. The MD_MAX7219 library allows you to escape ASCII codes, so I just need to find the two bytes where the symbol occurs, and replace them with the escaped value in the string.

I've tried a few different methods to convert the JSON object property to a character array, and they all result in the same behaviour. Specifically, the methods I've tried are:

sprintf(newMessage, root["message"]);
root["message"].printTo(newMessage, BUF_SIZE); // Includes open and close quotation marks.
snprintf(newMessage, BUF_SIZE, root["message"]);
root["message"].as<String>().getBytes( (unsigned char*)newMessage, BUF_SIZE);
root["message"].as<String>().toCharArray(newMessage, BUF_SIZE);
4

I've tried a few different methods to convert the JSON object property to a character array

You are starting with a char array. How is the degree symbol represented in that array? Print the array, one element at a time, as a character and the hex value of that character.

5

I've written out the values of each character to the serial terminal. The content was "test °" (without the quotes), and the serial monitor shows:

t 116 74
e 101 65
s 115 73
t 116 74
32 20
⸮ 194 C2
⸮ 176 B0

Clearly, there are two bytes there for the degree symbol, C2 and B0. How do I find those in the string (or character array), and replace them with "\248" (again, without quotes) - i.e. 4 characters.

Thanks for the help debugging. I'm not well versed in C, so I'm keen to improve!

6

You can use strchr() to find the character(s)

Simple demo

char buffer[10] = "test ";

void setup()
{
  Serial.begin(115200);
  while (!Serial);  // code written for a Leonardo as the target board
  Serial.println("ready");

  // add the strange characters
  buffer[5] = 0xC2;
  buffer[6] = 0xB0;

  // print
  Serial.println(buffer);

  // find the 0xB0
  char *ptr = strchr(buffer, 0xB0);

  // if found
  if (ptr != NULL)
  {
    // print a message
    Serial.println("0xB0 found");
    // replace
    *ptr = '!';
    // print updated buffer
    Serial.println(buffer);
  }
  // if not found
  else
  {
    // print a message
    Serial.println("0xB0 not found");
  }
}

void loop()
{
}

The above code will find the 0xB0 and replace it with '!'.

7

You are dealing with double byte characters. Please see how this is handled in the MD_Parola example "Parola_UFT-8_Display.ino".