URL converter into text

potzli · January 25, 2022, 2:16pm

Hello,

How to print corretly a String ("ALOE%20BLACC%20%26%20PEGASUS%20") into ALOE BLACC & PEGASUS ?

I haven't found a library.

Thanks

missdrew · January 25, 2022, 2:35pm

Write your own parser.

martinius96 · January 25, 2022, 2:37pm

Is it that hard? ESP8266 URL Encode Decode Example | Circuits4you.com

sherzaad · January 25, 2022, 3:20pm

something like this maybe:

void setup() {
  // put your setup code here, to run once:
  char *percent_enc[34] = {"%20", "%22", "%25", "%2D", "%2E", "%3C", "%3E", "%5C", "%5E", "%5F", "%60", "%7B", "%7C", "%7D", "%7E", "%21", "%23", "%24", "%25", "%26", "%27", "%28", "%29", "%2A", "%2B", "%2C", "%2F", "%3A", "%3B", "%3D", "%3F", "%40", "%5B", "%5D"};
  char *url_decoded[34] = {" ", "\"", "%", "-", ".", "<", ">", "\\", "^", "_", "`", "{", "|", "}", "~", "!", "#", "$", "%", "&", "'", "(", ")", "*", "+", ",", "/", ":", ";", "=", "?", "@", "[", "]"};
  char * pch;
  char str[] = "ALOE%20BLACC%20%26%20PEGASUS%20";

  Serial.begin(115200);

  Serial.print( "Before Decoding: ");
  Serial.println(str);

  //iterate through the percent encoding array
  for (int i = 0; i < 34; ++i) {
    //find first percent url encoding in string
    pch = strstr (str, percent_enc[i]);
    while ( pch != NULL ) {
      //replace url encoding with corresponding character
      strncpy (pch, url_decoded[i], strlen(url_decoded[i]));

      //remove any remaining characters that were part of the url encoding
      memmove(pch + strlen(url_decoded[i]), pch + strlen(percent_enc[i]), strlen(pch + strlen(url_decoded[i])));

      //find next percent url encoding in string
      pch = strstr (str, percent_enc[i]);
    }
  }

  Serial.print( "After Decoding: ");
  Serial.println(str);


}

void loop() {
  // put your main code here, to run repeatedly:

}

Hope that helps...

potzli · January 25, 2022, 3:51pm

Yes perfect, I just need to convert the String into char!! Thanks a lot!

sherzaad · January 25, 2022, 3:56pm

there's already a function for that!

sterretje · January 25, 2022, 4:01pm

But %20 will result in 3 characters instead of a space

potzli · January 25, 2022, 4:28pm

Is it easy to add accentued chars ?

éàü:
\xc3\xa9\xc3\xa0\xc3\xbc

because it needs "2 byte" for one char !

sherzaad · January 25, 2022, 4:53pm

not quite following what you mean there but I should think is just a matter of just extending the array in the example code to include those character codes.

"\\xc3\\xa9" for "é" for example.

also please be aware that " \ " should be expressed a " \\ " to be read as the backslash character.

hope that helps...

potzli · January 25, 2022, 5:06pm

Yes exactely !

I would write: é = %C3%A9 not slash !

potzli · January 28, 2022, 8:19am

Any idea how to add double (éàü) /triple (€) coded char like:

%C3%A9%C3%A0%C3%BC%E2%82%AC = éàü€
Thanks

sherzaad · January 28, 2022, 9:16am

did you try adding the respective 'url codes' to the array of the example code as previously suggested?

I would put longer 'url codes' as the beginning of the array, to reduce the risk of misreading which checking the actual url string.

e.g "%3C" -> ''
%C3%A9% -> 'é' //make sure this code is before "%3C" in array since contains the same character

hope that helps...

potzli · January 28, 2022, 10:09am

I'm not sure to add it correctly.

 char *percent_enc[36] = {"%20", "%22", "%25", "%2D", "%2E","%C3%A8" "%3C", "%3E", "%5C", "%5E", "%5F", "%60", "%7B", "%7C", "%7D", "%7E", "%21", "%23", "%24", "%25", "%26", "%27", "%28", "%29", "%2A", "%2B", "%2C", "%2F", "%3A", "%3B", "%3D", "%3F", "%40", "%5B", "%5D", "%C3"};
  char *url_decoded[36] = {" ", "\"", "%", "-", ".", "è", "<", ">", "\\", "^", "_", "`", "{", "|", "}", "~", "!", "#", "$", "%", "&", "'", "(", ")", "*", "+", ",", "/", ":", ";", "=", "?", "@", "[", "]", "é"};
  char * pch;

With:
char str[] = "Premi%C3%A8re";
Before Decoding: Premi%C3%A8re
After Decoding: Premi]%A8re
Instead of Première

sherzaad · January 28, 2022, 11:02am

works fine for me!

#define URL_SET 39

void setup() {
  // put your setup code here, to run once:
  struct {
    char const *percent_enc;
    char const *percent_dec;
  } c[URL_SET] = {{"%E2%82%AC", "€"}, {"%C3%A8", "è"}, {"%C3%A9", "é"}, {"%C3%A0", "à"}, {"%C3%BC", "ü"}, {"%20", " "}, { "%22", "\""}, { "%25", "%"}, { "%2D", "-"}, { "%2E", "."}, { "%3C", "<"}, { "%3E", ">"}, { "%5C", "\\"}, { "%5E", "^"}, { "%5F", "_"}, { "%60", "`"}, { "%7B", "{"}, { "%7C", "|"}, { "%7D", "}"}, { "%7E", "~"}, { "%21", "!"}, { "%23", "#"}, { "%24", "$"}, { "%25", "%"}, { "%26", "&"}, { "%27", "'"}, { "%28", "("}, { "%29", ")"}, { "%2A", "*"}, { "%2B", "+"}, { "%2C", ","}, { "%2F", " /"}, { "%3A", ":"}, { "%3B", ";"}, { "%3D", "="}, { "%3F", "?"}, { "%40", "@"}, { "%5B", "["}, { "%5D", "]"}};
  char * pch;

  //char test_str[] = "ALOE%20BLACC%20%26%20PEGASUS%20";
  char test_str[] = "Premi%C3%A8re%20%20%C3%A9%C3%A0%C3%BC%E2%82%AC";

  Serial.begin(115200);

  Serial.print( "Before Decoding: ");
  Serial.println(test_str);

  //iterate through the percent encoding array
  for (int i = 0; i < URL_SET; ++i) {
    //find first percent url encoding in string
    pch = strstr (test_str, c[i].percent_enc);
    while ( pch != NULL ) {
      //replace url encoding with corresponding character
      strncpy (pch, c[i].percent_dec, strlen(c[i].percent_dec));

      //remove any remaining characters that were part of the url encoding
      memmove(pch + strlen(c[i].percent_dec), pch + strlen(c[i].percent_enc), strlen(pch + strlen(c[i].percent_dec)));

      //find next percent url encoding in string
      pch = strstr (test_str, c[i].percent_enc);
    }
  }

  Serial.print( "After Decoding: ");
  Serial.println(test_str);


}

void loop() {
  // put your main code here, to run repeatedly:

}

Before Decoding: Premi%C3%A8re%20%20%C3%A9%C3%A0%C3%BC%E2%82%AC
After Decoding: Première éàü€

potzli · January 28, 2022, 2:28pm

For me too !!! Thanks a lot sherzaad !!!

potzli · February 7, 2022, 8:02pm

Perfect sherzaad!

i just noticed if the text is long it will be cut. Where is the limitation ?

Try with:

char str[] = "Dr%C3%B4le%20d'%C3%A9poque%20%3A%20Nahum%20Frenck%2C%20p%C3%A9diatre%20et%20th%C3%A9rapeute%20de%20famille%3B%20Dr%C3%B4le%20d'%C3%A9poque%20%3A%20Nahum%20Frenck%2C%20p%C3%A9diatre%20et%20th%C3%A9rapeute%20de%20famille%3B";

Thanks

sherzaad · February 7, 2022, 9:43pm

not sure what you mean...

#define URL_SET 40

void setup() {
  // put your setup code here, to run once:
  struct {
    char const *percent_enc;
    char const  *percent_dec;
  } c[URL_SET] = {{"%E2%82%AC", "€"}, {"%C3%A8", "è"}, {"%C3%A9", "é"}, {"%C3%A0", "à"}, {"%C3%B4", "ô"}, {"%C3%BC", "ü"}, {"%20", " "}, { "%22", "\""}, { "%25", "%"}, { "%2D", "-"}, { "%2E", "."}, { "%3C", "<"}, { "%3E", ">"}, { "%5C", "\\"}, { "%5E", "^"}, { "%5F", "_"}, { "%60", "`"}, { "%7B", "{"}, { "%7C", "|"}, { "%7D", "}"}, { "%7E", "~"}, { "%21", "!"}, { "%23", "#"}, { "%24", "$"}, { "%25", "%"}, { "%26", "&"}, { "%27", "'"}, { "%28", "("}, { "%29", ")"}, { "%2A", "*"}, { "%2B", "+"}, { "%2C", ","}, { "%2F", " /"}, { "%3A", ":"}, { "%3B", ";"}, { "%3D", "="}, { "%3F", "?"}, { "%40", "@"}, { "%5B", "["}, { "%5D", "]"}};
  char * pch;

  //char test_str[] = "ALOE%20BLACC%20%26%20PEGASUS%20";
  //char test_str[] = "Premi%C3%A8re%20%20%C3%A9%C3%A0%C3%BC%E2%82%AC";
  char test_str[] = "Dr%C3%B4le%20d'%C3%A9poque%20%3A%20Nahum%20Frenck%2C%20p%C3%A9diatre%20et%20th%C3%A9rapeute%20de%20famille%3B%20Dr%C3%B4le%20d'%C3%A9poque%20%3A%20Nahum%20Frenck%2C%20p%C3%A9diatre%20et%20th%C3%A9rapeute%20de%20famille%3B";
  Serial.begin(115200);

  //Serial.print( "Before Decoding: ");
  //Serial.println(test_str);

  //iterate through the percent encoding array
  for (int i = 0; i < URL_SET; ++i) {
    //find first percent url encoding in string
    pch = strstr (test_str, c[i].percent_enc);
    while ( pch != NULL ) {
      //replace url encoding with corresponding character
      strncpy (pch, c[i].percent_dec, strlen(c[i].percent_dec));

      //remove any remaining characters that were part of the url encoding
      memmove(pch + strlen(c[i].percent_dec), pch + strlen(c[i].percent_enc), strlen(pch + strlen(c[i].percent_dec)));

      //find next percent url encoding in string
      pch = strstr (test_str, c[i].percent_enc);
    }
  }

  Serial.print( "After Decoding: ");
  Serial.println(test_str);


}

void loop() {
  // put your main code here, to run repeatedly:

}

Drôle d'époque : Nahum Frenck, pédiatre et thérapeute de famille; Drôle d'époque : Nahum Frenck, pédiatre et thérapeute de famille;

bear in mind that this code can be quite memory intensive and if you create too long strings and/or decoding sets, you may very well run out of memory!

potzli · February 8, 2022, 4:38pm

Thanks sherzaad!

After some tests (downloading the code many times). Your code is working well but sometimes the RS232 output is truncked (always at the same place:
After Decoding: Drôle d'époque : Nahum Frenck, pédiatre et thérapeute de famille; Drôle d'époque : Nahum Frenck, pédiatre et thérapeute de famil), rarely the output is complete!!! Tried with lower/higher baud rate, no change. Added Serial.flush(); no change.
The problem is on the serial side! (I'm Using Teensy 3.2)

I tried to expand the serial buffer: https://forum.pjrc.com/threads/54773-Inreasing-USB-Serial-Buffer-Teensy-3-2?p=194517&viewfull=1#post194517

no change ! Strange !!!

sherzaad · February 8, 2022, 4:47pm

the issue seems hardware specific then...

I'm using a uno and so far its be outputing correctly. not really familiar with the teeny series unfortunately.

potzli · February 15, 2022, 8:45pm

updated IDE and it works.

With the new one 1.8.19 some warnigs appears
warning: ISO C++ forbids converting a string constant to 'char*'

But until now, it works.

Topic		Replies	Views
esp8266 read webpage information Programming	12	5457	May 5, 2021
spaces in url request Programming	4	1599	June 5, 2021
URL encode char* Programming	23	23787	May 5, 2021
HTTP GET Requests ESP8266 Programming	5	4483	May 5, 2021
URL Querystring parsing and decoding Showcase	1	4545	May 6, 2021

URL converter into text

Related topics