Tokenizing a string

Using Arduino Programming Questions
#1

Hi,

I am receiving a data serially of the format

AAAA;A1:3;B1:5;ZZZZ

I want to be able to get the A1, 3, B1, 5 separated from this. I tried using strtok to separate AAAA, A1:3, B1:5 & ZZZZ and I was successful. I am not able to separate A1:3 and the B1:5 part using the same strtok with the ':' delimiter. Any suggestions on how to proceed with this problem?

Cheers!

#2

This code works like you want, I think. Hard to say what you were doing wrong as you posted no code.

char array[] = "AAAA;A1:3;B1:5;ZZZZ";
char *strings[10];
char *ptr = NULL;

void setup()
{
    Serial.begin(9600);
    //Serial.print(array);
    byte index = 0;
    ptr = strtok(array, ":;");  // takes a list of delimiters
    while(ptr != NULL)
    {
        strings[index] = ptr;
        index++;
        ptr = strtok(NULL, ":;");  // takes a list of delimiters
    }
    //Serial.println(index);
// print the tokens
    for(int n = 0; n < index; n++)
   { 
    Serial.println(strings[n]);
   }
}

void loop()
{
    // put your main code here, to run repeatedly:

}

Output:

AAAA
A1
3
B1
5
ZZZZ

#3

abhir24:
using the same strtok with the ':' delimiter. Any suggestions on how to proceed with this problem?

The strtok() function does NOT take a single character like ':' as a delimiter.

strtok() uses a delimiter string (nullterminated string), so that you can have different delimiter characters like ":;" instead of only one character.

#4

groundfungus:
This code works like you want, I think. Hard to say what you were doing wrong as you posted no code.

Thank you, groundfungus & @jurs! The code by @groundfungus certainly works for the problem at hand! I didn't know that I could give a list of delimiters to the strtok function. And, I tried splitting the string into tokens using the ; and then splitting the ones with the : into 2 by passing it into another function which was written just for that.

#5

groundfungus:
This code works like you want, I think. Hard to say what you were doing wrong as you posted no code.

I dont know if I am understanding the code correctly and there is a problem I am facing.

The following is my code

if(strings[0] == "AAAA")
Serial.println('s');
else 
Serial.println('n');

I want to find a certain string in that and act accordingly. How can I do that?

I realize that char *strings[10] are pointers to where the split strings are located. How can I use the values in the strings for something like the above?

#6

Who's allocating the memory for the output token strings? strtok()? Who de-allocates it?

-jim lee

#7

jimLee:
Who's allocating the memory for the output token strings? strtok()? Who de-allocates it?

-jim lee

I am not sure of any of these answers. I am new to strings and the functions that work on strings.
I need to read more, I think.

#8 
char array[] = "AAAA;A1:3;B1:5;ZZZZ";
char *strings[10];
char *ptr = NULL;

void setup()
{
    Serial.begin(9600);
    Serial.println(array);
    byte index = 0;
    ptr = strtok(array, ":;");  // takes a list of delimiters
    while(ptr != NULL)
    {
        strings[index] = ptr;
        index++;
        ptr = strtok(NULL, ":;");  // takes a list of delimiters
    }
    //Serial.println(index);
// print the tokens
    for(int n = 0; n < index; n++)
   { 
    //Serial.println(strings[n]);
    int k = strlen(strings[n]);
    char data[k];
    for(int A = 0; A < k; A++)
    {
     data[A] = *strings[n];
     *strings[n]++;
    }
    Serial.println(data);
   }
}

void loop()
{
    // put your main code here, to run repeatedly:

}

After a bit of reading, I coded this and it solved my purpose! Thank you!

#9

Who's allocating the memory for the output token strings?

No extra memory is needed. strtok() destroys the original string, by putting NULLs in place of the tokens, returning pointers to the appropriate parts of the original memory.

Who de-allocates it?

Since nothing was allocated, nothing needs to be freed.

It would often be nice to be able to tokenize a string, like "AAAA;A1:3;B1:5;ZZZZ" and get the tokens "AAAA", "A1:3", "B1:5", and "ZZZZ", and then be able to tokenize the tokens that contain colons independently, but, alas, that is not possible without making copies of those tokens and parsing them separately.

It's also not possible to have strtok() tell you which delimiter that it found as the next delimiter.

So, you really need to know a lot about the string that you are trying to parse.

#10

Thank you for the insight, @PaulS. I do know the specifics of what I am going to receive, so it is helpful. Like you said, if I didn't know about it, it would have a been a different problem to solve altogether!

#11