Go Down

Topic: convert GSM 7-bit encoded text to utf-8 (Read 206 times) previous topic - next topic

pojamapeoples

Hi,

I'm receiving sms messages using the Adafruit Fona GSM module. My problem is the character encoding of the SMS messages. The appear to come in a 7-bit encoding and I need them in UTF-8. This gives problems with high ascii character like the German "ö".

Someone has a method to convert the encoding?

Thanks!

sterretje

No experience with this but (based on the tables in the link you provided and my understanding of the link on how it works) you might be able to solve this with a lookup table; it will consume some memory.

Code: [Select]
byte lookup[][2] =
{
  {0x40, 0x00},
  {0xC2, 0xA3},
  {0x24, 0x00},
  ...
  ...
}

GSM character 0 is the index of the first element in the array, GSM character 1 is the index of the second element in the array and so on.
The value 0x00 indicates the character does not need to be printed.

Note that the table must be sequential and you can't have missing entries. E.g. an entry for the escape character must be there; it can be {0x00, 0x00}.

In the  below, gsmChar is the received character
Code: [Select]

if(lookup[gsmChar][0] != 0x00)
{
  // send first byte
  Serial.print(lookup[gsmChar][0]);
}

if(lookup[gsmChar][1] != 0x00)
{
  // send second byte
  Serial.print(lookup[gsmChar][1]);
}

If you want to be able to handle extended gsm characters, you will need to check if 0x1B is received; easiest might be to handle this with a switch/case.
Code: [Select]

if(gsmChar!=0x1B)
{
  // indicate extended mode
  mode = 1;
}

if(mode==0)
{
  if(lookup[gsmChar][0] != 0x00)
  {
    // send first byte
    Serial.print(lookup[gsmChar][0]);
  }

  if(lookup[gsmChar][1] != 0x00)
  {
    // send second byte
    Serial.print(lookup[gsmChar][1]);
  }
}
else
{
  // handle extended gsm characters
  ...
  ...
  // after handling extended characters, reset mode no normal
  mode = 0;
}

The above demonstrates the idea, not compiled or tested. If you can show your code, you can get a more tailored advise.
If you understand an example, use it.
If you don't understand an example, don't use it.

Electronics engineer by trade, software engineer by profession. Trying to get back into electronics after 15 years absence.

pojamapeoples

@sterretje thanks for the quick reply. Things are even easier for me - I don't need to cover all high ascii values, but only 7 chars:

(öäüÖÄÜß). I ignore all others.

I'm still confused about the values I received. My test SMS contained "mit mehr öäü". That should be:

{6D,69,74,20,6D,65,68,72,20...} and now three high ascii value {7C,7B,7E} according to the GSM encoding table I linked before.

with this little method:

Code: [Select]

void printASCIICode(char * myString) {
  char currentChar;

  for(int i=0;i<strlen(myString);i++) {
    currentChar = myString[i];
    Serial.println(currentChar,HEX);
  }
}


I got this:

6D
69
74
20
6D
65
68
72
20
F6
E4
FC

Why do I get this high numbers (dec 246, 228 and 252) for the chars that should be 7C,7B,7E?

The relevant part of my code to retrieve the SMS:
Code: [Select]

    //any data available from the FONA?
    do  {
      //Read the notification into fonaNotificationBuffer
      *bufPtr = fona.read();
      delay(1);
    } while ((*bufPtr++ != '\n') && (fona.available()) && (++charCount < (sizeof(fonaNotificationBuffer)-1)));

    *bufPtr = 0; //Add a terminal NULL to the notification string

    if (1 == sscanf(fonaNotificationBuffer, "%*c%*c+CMTI: " FONA_PREF_SMS_STORAGE ",%d", &slot)) {

      // Retrieve SMS sender phone number.
      if (! fona.getSMSSender(slot, callerIDbuffer, 31)) {
        Serial.print(F("Didn't find SMS message in slot 1: "));
        MYDEBUG_PRINTLN(slot);
      }
      MYDEBUG_PRINT(F("FROM: ")); MYDEBUG_PRINTLN(callerIDbuffer);

      // Retrieve SMS value.
      if (fona.readSMS(slot, smsBuffer, 250, &smslen)) { // pass in buffer and max len!
        strcat(msgText, smsBuffer); // the msg
        strcat(msgText, "#");
        strcat(msgText, callerIDbuffer); // the callers number
        emptyCharBuffer();

        // delete the original msg after it is processed
        deleteSingleSMS(slot);
      }


pojamapeoples

#3
Apr 22, 2019, 04:59 pm Last Edit: Apr 22, 2019, 05:04 pm by pojamapeoples
shouldn't this method do the job (it does not - no change to see in the message):

Code: [Select]

void printASCIICode(char * myString) {
  for(unsigned i=0;i<strlen(myString);i++) {
    Serial.println(myString[i],HEX);
    if(myString[i] == 0xf6) {
      Serial.println("Replace");
      myString[i] = 246; // oe = 0xc3,0xb6
    }
  }
}


Oh - just saw that 0xf6 = 246... so I'm replacing the character with an identical value. Smart...
But why does the original character comes as 0xf6 and does not display as "ö" and if I check the hex value of an UTF8 "ö" I get 0xf6 too but this this time it will be displayed?!

pojamapeoples

anyone has an idea why the received char with the value 0xf6 will not be displayed as "ö" in the serial monitor or on the connected display while an "ö" I passed in the code will return the same hex value and will be displayed in the serial monitor and the display??? This should be a very basic thing I just don't get it... :(

Go Up