Refactor strtok to separate function

Im trying to refactor out the strtok functionality to a separate reusable function but facing problems with pointers.

The following code works:

void loop() {
  static char buffer[160];
  if (readline(simSerial->read(), buffer, 160) > 0) {
    Serial.print(">");
    Serial.print(buffer);
    Serial.println("<");
    if (strncmp(buffer, "######", 6) == 0) {


      // Start of strtok
      char *token = strtok(buffer, ",");
      char *tokens[2];
      int i = 0;
      while (token != NULL) {
        tokens[i++] = token;
        token = strtok(NULL, ",");
      }
      // End strtok

      Serial.println(tokens[1]);
    }
  }
}

I would like to be able to change it to something like:

void loop() {
  static char buffer[160];
  if (readline(simSerial->read(), buffer, 160) > 0) {
    Serial.print(">");
    Serial.print(buffer);
    Serial.println("<");
    if (strncmp(buffer, "######", 6) == 0) {


      // Refactored
      char delimiter = ",";
      char tokens = getStringTokens(buffer, delimiter);

      Serial.println(tokens[1]);
    }
  }
}

What would the function getStringTokens look like?

Since tokens is an array, you either need to make it be global or pass a pointer to it as an argument. You can't return more than one value from a function, so you can't return an array. Since there will be more than one token, returning them is out of the question.

// Refactored
      char delimiter = ',';         ////  THIS SHOULD BE SINGLE QUOTES 
      char* tokens[2];
      getStringTokens(tokens, buffer, delimiter);

and your function will look something like:

void getStringTokens(char** tokens, char* buffer, char delimiter){
      char *token = strtok(buffer, ",");
      int i = 0;
      while (token != NULL) {
        tokens[i++] = token;
        token = strtok(NULL, ",");
      }

Thank you that works brilliantly!

Why the double ** when declaring input parameter char** tokens?
Quite new to c.

Because tokens is an array of pointers to char. So a pointer to that would be a pointer to a pointer to a char. Ergo the double star, pointer to a pointer.

Thanks!

@Delta-G: would it be useful to make the delimiter variable an array and then use it in the new function, perhaps like:

void getStringTokens(char** tokens, char* buffer, char delimiter[]){
      char *token = strtok(buffer, delimiter);
      int i = 0;
      while (token != NULL) {
        tokens[i++] = token;
        token = strtok(NULL, delimiter);
      }

I think that is actually what strtok is expecting.

In the OPs code I corrected to use single quotes because it was being assigned to a char variable. But in reality, strtok takes a const char* IIRC.

In that case I think you’re right but I would write char* instead of char.

void getStringTokens(char** tokens, char* buffer, char* delimiter){
      char *token = strtok(buffer, delimiter);
      int i = 0;
      while (token != NULL) {
        tokens[i++] = token;
        token = strtok(NULL, delimiter);
      }

This should ideally also include the size of tokens and check to make sure you don’t write off the end of that array.

void getStringTokens(int maxTokens, char** tokens, char* buffer, char* delimiter){
      char *token = strtok(buffer, delimiter);
      int i = 0;
      while ((token != NULL) && (i < maxTokens) ){
        tokens[i++] = token;
        token = strtok(NULL, delimiter);
      }

Then you could pass the size of the tokens array and make sure you don’t go too far on it.

@Delta-G: Good idea to pass in the size of tokens so you don't write past the end of the array. Also, when I post here, I tend to use char delimiters[] instead of char *delimiters even though they are the same. For some reason, the pointer operator causes some beginners to freeze in their tracks but they're okay with the array operator.