Converting Hex value to Ascii to decimal more efficiently

I've been scouring these forums for the last few days to solve my problem which I've finally done. However i feel like I'm not doing it very efficiently and am looking for guidance.

Background:
I have an arduino uno connected to a modem via serial. Periodically the Uno queries the modem with some AT commands. The modem has a unique identifier (ISSI) that i want to get and have written a function for below.

void getISSI() {
  //Modem command to retrieve registration information  
  String cmd = "AT+CNUM?\r\n";
  Serial.print(cmd);
  serData = Serial.readString();
  // Where ISSI is located in modem output, is returned as HEX(in ASCII)
  serData = serData.substring(20, 24);
  //setup char array
  int str_len = serData.length() + 1;
  char ISSI[str_len];
  //Put String into char array
  serData.toCharArray(ISSI, str_len);
  //Convert to char array to unsigned long
  int ID;
  ID = strtoul(ISSI, NULL, 16);
  
  monSerial.print(ID);
}

When querying the modem with the AT+CNUM command this is the output:

AT+CNUM?<cr>
<cr><lf>+CNUM: 0,3f4<cr><lf>
0,0<cr><lf>
1,16777215<cr><lf>

3f4 is the sub-string I'm extracting, but it's a String, and i want to convert that hex string to its decimal value (1012). majority of what i have read suggests converting to a long which I've done, or avoid using Strings all together but I don't have the experience to know how to do that...

Is there a better way to achieve my desired end result? This is only one function in what will likely have many similar functions and I'm afraid of the amount of memory resources that will ultimately be used.

Thanks

This is only one function in what will likely have many similar functions

Your function does too much. It should get the response, and pass it to a function, along with the start and end positions of the data of interest. That function should create the substring, extract the characters, and convert them to an unsigned long, which it should return.

Your one function should become two functions, one of which can be used over and over.

PaulS:
Your function does too much. It should get the response, and pass it to a function, along with the start and end positions of the data of interest. That function should create the substring, extract the characters, and convert them to an unsigned long, which it should return.

Your one function should become two functions, one of which can be used over and over.

Ok, i get that this single function is doing too much. It's more or less because I was trying to muddle my way through getting the desired result via trial and error.

I guess what I'm really looking for here, is there a better way to deal with the serial output of the modem than what I'm doing? I'm reading everything in as a String variable, the modem will always output ascii, but sometimes the value i want is represented in hex, other times decimal. For the sake of conversion simplicity and efficiency should i just read it into an array in the first place?

Generally, you would read one line at a time by reading the characters into a char[] until you hit the cr/lf. Discard the cr/lf. ONe way to do this is using the strtok() function.

As you get each line, pull out the line you want - either search each line for the text "CNUM: " using the strstr() function, or just pull out the correct line (by keeping count).

Use a char pointer to point to the first character after the "CNUM: ".

Find the comma using strchr() or just by moving the pointer along.

If you have zapped the cr/lf with strtok, the the pointer will now be pointing at "34f" with a nul \0 terminator.

Use strtoi() or strtoul() as you are currently doing to get the number.

These various string manipulation functions are all described in the documentation for AVR libc.

Of course, the code is much more compact than this description.

Ok, i get that this single function is doing too much. It's more or less because I was trying to muddle my way through getting the desired result via trial and error.

That's fine. But, once the code is working, you should move it to its own function.

is there a better way to deal with the serial output of the modem than what I'm doing?

Yes. You could put the crutches (String class) away, and learn to use char arrays. The readString() method simply reads all available serial data and waits some default amount of time for there to be more. It adds the characters to a String.

You can, in 5 minutes, replicate the functionality using a NULL terminated array of chars. And, you should.

Then, you can simply pass strtoul() the address of the array position that contains the character to start at.

But, that's a lot of work for a newbie. So, the question is just how much work do you want to do to improve your program?

PaulS:
But, that's a lot of work for a newbie. So, the question is just how much work do you want to do to improve your program?

Everything I've read here about this says to avoid Strings so I want to. In this instance time is a limiting factor, but I don't want to go down a road where in the future I'm going to have to turn around and make the changes you said anyways because I've hit a wall. I'm not afraid of work but it'll be more than 5 minutes for me.

Electronics is my thing, always struggled with programming in school (c specifically)... Recently i picked up an rPi and started with python and for some reason many things made sense that didn't before. But I also didn't have to think much about variable types since python takes care of that for you, and type conversion is pretty simple. Could also be an age thing... Who knows.

As far as reading the serial data into a char array. What if i don't know how many characters I'm going to receive? should i just declare a large enough array that can accommodate whatever i may read? I don't know what best/typical practices are.

but I don't want to go down a road where in the future I'm going to have to turn around and make the changes you said anyways because I've hit a wall.

You are already going down that road.

Robin2 has a great tutorial on how to read serial data with nary a String in sight.
http://forum.arduino.cc/index.php?topic=396450.0

and for some reason many things made sense that didn't before.

Not surprising. It's like looking at one side of a piece of art, and saying "I don't get it", and then walking back by from a different angle, and suddenly getting it.

Could also be an age thing... Who knows.

For those making an issue of age, we have 12 to 80 year olds on this forum, so age is an issue only if you are outside that range. I'm 60, and have no problems with programming. Been doing it for more than 40 years now, and loving every minute of it. Well, almost every minute...

What if i don't know how many characters I'm going to receive?

You'll have to decide on a maximum that you want to deal with. 80 is common upper limit. From the days of punch cards...

should i just declare a large enough array that can accommodate whatever i may read?

Yes. Don't go overboard thinking you need to deal with 512 character messages, though.

I don't know what best/typical practices are.

Personally, I wouldn't worry too much about best practices, yet. Get a few successful projects completed, and what works FOR YOU will be a "best practice".

PaulMurrayCbr:
Generally, you would read one line at a time by reading the characters into a char[] until you hit the cr/lf. Discard the cr/lf. ONe way to do this is using the strtok() function.

As you get each line, pull out the line you want - either search each line for the text "CNUM: " using the strstr() function, or just pull out the correct line (by keeping count).

Use a char pointer to point to the first character after the "CNUM: ".

Find the comma using strchr() or just by moving the pointer along.

If you have zapped the cr/lf with strtok, the the pointer will now be pointing at "34f" with a nul \0 terminator.

Use strtoi() or strtoul() as you are currently doing to get the number.

These various string manipulation functions are all described in the documentation for AVR libc.

Of course, the code is much more compact than this description.

Thank you for the direction. I believe i have been able to utilize the functions you specified, code to follow.

PaulS:
For those making an issue of age, we have 12 to 80 year olds on this forum, so age is an issue only if you are outside that range.

Heh, i was implying that my attitude towards programming in my youth was poor. likely because i wasn't grasping key concepts very well i decided early on that i didn't need to learn it... boy was i wrong.

Anyways, thank you both for the help. I've written two functions, one to read serial data into a char array, stripping the and then passing it to a function to parse. no Strings :slight_smile:

as for the parser it's a work in progress so be gentle... I just wanted to prove that i could use strstr() and strrchr(). My variable usage may not be efficient, so please pipe in if I'm doing something completely wrong here. Also ignore the checkConn() function, I'm going to redo it but was the first i wrote for this project so it's still in the sketch.

#include <SoftwareSerial.h>

SoftwareSerial monSerial(10, 11); //Rx, Tx

void setup() {
  Serial.begin(38400);
  Serial.setTimeout(500);
  monSerial.begin(38400);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
}

void loop() {  
  checkConn();
  Serial.print("AT+CNUM?\r\n");
  delay(100);
  readSer();
  delay(5000);
}

void readSer() {
  
  const int buffSize = 50; //Set buffer size
  char sData[buffSize]; //Array for storing serial data
  char sChar; //Character read from serial
  int len =0; //length of what's in buffer
  
  while(Serial.available() > 0) //only read if data
  {
    sChar = Serial.read(); //Read it
    if ((sChar == '\r') || (sChar == '\n')) {
      //EOL
      if (len > 0) {
        sParse(sData);
      }
      len = 0;
    }
    else {
      if (len < buffSize) {
        sData[len++] = sChar; //append recieved char to array
        sData[len] = '\0'; // append null terminator to end
      }
    }
  }
}

void sParse(char *msg) {
  const char * CNUM = "+CNUM: 0,";
  char * p;
  int ID;
  p = strstr(msg, CNUM);
  if (p) {
    char *ISSI;
    ISSI = strrchr(p, ',');
    ISSI++;
    ID = strtoul(ISSI, NULL, 16);
    monSerial.println(ID);
  } 
}

void checkConn() {
  String serData;
  digitalWrite(LED_BUILTIN, LOW);
  int serConn = 0;
  while (serConn ==0) {
    Serial.print("AT\r\n");
    delay(100);
    serData = Serial.readString();
    //monSerial.print(serData);
    
    if (serData.indexOf("OK") > 0) {
      //Set LED on
      digitalWrite(LED_BUILTIN, HIGH);
      //Put modem in advanced mode
      Serial.print("AT+PIMODE=1\r\n");
      serData = Serial.readString();
      //monSerial.print(serData);
      serConn = 1;
    }
  }
}

i decided early on that i didn't need to learn it... boy was i wrong.

Things change. You can embrace the change or you can fight it.

no Strings

Great.

void checkConn() {
  String serData;

Uh-oh.

PaulS:

void checkConn() {

String serData;



Uh-oh.

kmarts:
... Also ignore the checkConn() function, I'm going to redo it but was the first i wrote for this project so it's still in the sketch.

:stuck_out_tongue:

kmarts:
:stuck_out_tongue:

Oops. Stopped reading too early. Sorry.

PaulMurrayCbr:
Generally, you would read one line at a time by reading the characters into a char[] until you hit the cr/lf. Discard the cr/lf. ONe way to do this is using the strtok() function.

As you get each line, pull out the line you want - either search each line for the text "CNUM: " using the strstr() function, or just pull out the correct line (by keeping count).

Use a char pointer to point to the first character after the "CNUM: ".

Find the comma using strchr() or just by moving the pointer along.

If you have zapped the cr/lf with strtok, the the pointer will now be pointing at "34f" with a nul \0 terminator.

Use strtoi() or strtoul() as you are currently doing to get the number.

These various string manipulation functions are all described in the documentation for AVR libc.

Of course, the code is much more compact than this description.

I may be deviating from the original question here a bit but since it's all related to the same sketch I'm hoping it's ok to continue asking questions in this thread?

Anywho, everything is working great and now I'm wanting to expand the functionality of the sParse() function to look for different replies to different commands and perform different tasks depending on what it recieves.

In my previous example i used strstr() to find the +CNUM sub-string which identifies the command response. Eventually I'm going to have several different responses i.e. +CSQ, +CTSDSR, etc...

looking at strstr() it requires a constant char type, from what I've read you're either not supposed to or can't change the contents of a constant char variable.

Does that mean i will have to create a variable for each substring i may want to look for? Is there a better/alternate way? It seems like that would start to chew up SRAM

looking at strstr() it requires a constant char type, from what I've read you're either not supposed to or can't change the contents of a constant char variable.

Can't is correct.

But, what const means in the call to strstr() is that strstr() won't modify that particular argument. It does NOT mean that what you pass to the function has to be const.

kmarts:
looking at strstr() it requires a constant char type, from what I've read you're either not supposed to or can't change the contents of a constant char variable.

It's the other way around: it forces the strstr function itself to not change the contents. That is: declaring a function argument to be const char* is a way of promising the person that uses the function that the function will not alter the string that is passed into it.

PaulS:
Can't is correct.

But, what const means in the call to strstr() is that strstr() won't modify that particular argument. It does NOT mean that what you pass to the function has to be const.

PaulMurrayCbr:
It's the other way around: it forces the strstr function itself to not change the contents. That is: declaring a function argument to be const char* is a way of promising the person that uses the function that the function will not alter the string that is passed into it.

You're both really helping me to understand this stuff. Thanks a lot.

PaulMurrayCbr:
That is: declaring a function argument to be const char* is a way of promising the person that uses the function that the function will not alter the string that is passed into it.

To continue:

This was part of a movement a few of decades back related to the idea of provability in code, and harks back to the notion of functional programming.

Certain expressions in C have "side effects". For instance, the assignment expression: a = b . = is a binary operator, like + or *. a = b is simply an expression has the value of whatever in in b. However, it has the side effect of changing the value of a. The issue is that this kind of thing is quite difficult to treat mathematically: which is kinda what you would expect. A mathematical theorem doesn't have state. Code does.

There's quite a bit of theory relating to all this. "side effect", "idempotent", "referential transparency", "reentrancy", "programming by contract" and so on. "const" and "volatile" are C language constructs dealing with some of those issues, as is broader things like the notion of a stack containing local variables.

The thing is: side effects cascade. Anything has a subexpression with a side-effect or that calls a function with a side-effect itself has side-effects. Potentially. This is significant for the whole problem of code analysis for purposes of testing, performance tuning, compiler optimization, that kind of thing. There's no way to determine if (and what) side-effects, what changes of state a chunk of code might perform without analyzing all of the functions that it potentially calls. The intent of the 'const' keyword is to cap this cascade.

For instance, if I pass a reference to an int variable to a function, that function might alter the variable. This means that the compiler must generate code to re-load the variable from memory after that function call completes. But if its a const reference, then the compiler is free to not have to do this. To enforce this, however, if you write a function and declare that one of the parameters is const, then that means that your function may not invoke other functions aainst that variable unless they, too, declare that that parameter is const. The promise not to alter the variable must propagate all the way down, or else the compiler will complain at you.

Personally, I dropped out of Uni back in the 80's and never got the full computer science education. Hasn't hindered me a lot, but it's still nice to have a general, nebulous kind of idea of what it's all kinda-sorta about.