Strtok works/doesn't depending upon where delimiter char is defined

Projects Programming
1

Hello,

I have a problem I don't understand, and would like to know if anyone has seen this before or can explain what is going on, as I'm stumped. The code below WORKS (i.e. the string rxString is parsed into tokens) if the char delim is defined within the function parseCommand(), but not if it is defined globally. I don't understand what the difference is.

Please forgive the nasty programming and structure, I'm experimenting and I thought I had everything on track until this showed up...

// Global variable declarations

String receivedTokens[20];
String commandString[6];
char rxString[] = "?e GO go 12838 -2032 0 +32.3 ? A1 500";  // text to tokenise
char* token;                                                // pointer to token
//char delim[] = { 32 };   // Decimal 32 is ASCII 'space' - FAILS if defined here


// ------------------------------------------------------------------------------------------------
// ------------------------------------------------------------------------------------------------
void setup() {
  Serial.begin(115200);
}

// ------------------------------------------------------------------------------------------------
// ------------------------------------------------------------------------------------------------
void parseCommands() {
   //    char rxString[] ="RL3]\[Test 1}|{3}|{S]\[Test 2}|{41}|{L]\[Test 3}|{1234}|{L";    // text to tokenise
  //char rxString[] = "?e GO go 12838 -2032 0 +32.3 ? A1 500";  // text to tokenise
  //char* token;                                                // pointer to token
  char delim[] = { 32 };    // Decimal 32 is ASCII 'space' - WORKS if defined here
  int i = 0;
  int validCommand;
  bool match = false;

  // Test some things out about String

  String tokenString[20];
  tokenString[0] = "Hello";
  tokenString[1] = "A3";

  //  String receivedTokens[20];

  Serial.print(tokenString[0]);
  Serial.print(tokenString[1]);
  Serial.println();

  // Store some commands in String array

  //  String commandString[6];
  commandString[0] = "A1";
  commandString[1] = "A2";
  commandString[2] = "A3";
  commandString[3] = "GO";
  commandString[4] = "?";


  // Moving on...

  Serial.print("Splitting rxString ");
  Serial.print(rxString);
  Serial.println(" into tokens:");

  token = strtok(rxString, delim);  // get first token
                                    //    token = strtok (rxString,"}]\[|{");         // get first token
  receivedTokens[i] = token;
  while (token != NULL) {
    Serial.print("Token found: ");
    Serial.println(token);        // print it
    token = strtok(NULL, delim);  // get next token
                                  //      token = strtok (NULL, " ");     // get next token
    i++;
    receivedTokens[i] = token;
  }

  // Show the tokens stored in the String array

  for (i = 0; i < 20; i++) {
    Serial.print(" receivedTokens[");
    Serial.print(i);
    Serial.print("]: ");
    Serial.println(receivedTokens[i]);
  }

  // Match the stored tokens against the 'acceptable commands'

  i = 0;
  while (!match && i < 5) {
    if (receivedTokens[0].equalsIgnoreCase(commandString[i])) {
      Serial.println("First token matches a command!");
      Serial.print(" receivedTokens[0]: ");
      Serial.print(receivedTokens[0]);
      Serial.println();
      Serial.print(" commandString[");
      Serial.print(i);
      Serial.print("]: ");
      Serial.print(commandString[i]);
      Serial.println();
      validCommand = i;
      match = true;
    } else {
      validCommand = -1;
      match = false;
    }
    i++;
  }
  if (validCommand == -1) {
    Serial.println(" ! Unrecognized command.");
    Serial.print("validCommand: ");
    Serial.print(validCommand);
    Serial.println();
  } else {
    Serial.print("validCommand: ");
    Serial.print(validCommand);
    Serial.println();
  }
}


// ------------------------------------------------------------------------------------------------
// ------------------------------------------------------------------------------------------------
void loop() {
parseCommands();
  while(true);
}
2

You were lucky it worked at all.

You are correct to think it should not matter where you declare and define the delimiter character array. You just did it wrong.

char delim[] = {32, 0};

should work placed at either location. I cannot test[ed] this just now. 99.44 100 percent sure.

Please read the documentation for strtok() at your favorite learning source. The delimiter character(s) need to be in a null-terminated character array.

It worked for you when it did because there happened to be a zero in the next byte of memory after your array of one element.

a7

3

The form more commonly seen:

char delim[] = " "; //space enclosed in double quotes - NOT single quotes
1 Like
4

Thank you - that does appear to be the problem. :grinning:

I was stupidly trying to be 'clever' by providing the ASCII code for the desired delimiter rather than the quoted string. Earlier, I had used the quoted string without problems, so my attempt to improve things (to my mind) broke it.

I appreciate your helpful replies.

5

Or even (possibly) better

const char * delim = " ";

6

@J-M-L - thanks for this. A few naive points/questions.

  • I sort-of see the point of the const modifier providing I don't want to change the delimiter at any point post-declaration. Is there any other reason to use it?

  • The char vs. pointer-to-char thing confuses me a bit. I get that a C-string is an array of characters (char values) plus a trailing NULL termination, but what's the rule/correct understanding for knowing when to declare something like this (delim here) as a simple char (as @david_2018 did) or as a pointer-to-char as you did?

8 
  char delimA[] = { 32 };

  char delimB[] = " ";

  const char * delimC = " ";

delimA, delimB and delimC are all pointers to char. delimB and delimC are pointers to null-terminated strings, delimA is not, your original problem.

C/C++ uses a very strong relationship between arrays and pointers.

a7

9

You usually define an array when you want to use the variable as such (like with an index going though the elements). Here we just want a c-string so the const char * feels more natural

As the others said, in the end it will work in the same way. So it’s more of a coding style habit than a strong established rule.

10

Is it delimiter character (as mentioned in the title of the thread) or delimiter string that is required by the strtok() function? For example:

char myString[] = "Arduino, Forum";
char *token;
token = strtok(myString, ",");

or

token = strtok(myString, ',');
11

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.