Strtok question

Hi,

My background is not in C (it's in Real Studio - similar to VB) and I'm really struggling to split a comma-delimited string since I'm not used to low-level string handling.

I'm sending strings to the Arduino over serial. These strings are commands in a certain format. For instance:

@20,2000,5! @10,423,0!

'@' is the header indicating a new command and '!' is the terminating footer marking the end of a command. The first integer after '@' is the command id and the remaining integers are data (the number of integers passed as data may be anywhere from 0 - 10 integers).

I've written a sketch that gets the command (stripped of the '@' and '!') and calls a function called handleCommand() when there is a command to handle. The problem is, I really don't know how to split this command up to handle it!

Here's the sketch code:

String command; // a string to hold the incoming command
boolean commandReceived = false; // whether the command has been received in full

void setup() {
  // put your setup code here, to run once:
Serial.begin(9600);
}

void loop() {
  // main loop
  handleCommand();
}

void serialEvent(){
  while (Serial.available()) {
  // all we do is construct the incoming command to be handled in the main loop
    
    // get the incoming byte from the serial stream
    char incomingByte = (char)Serial.read();
    
    if (incomingByte == '!')
    {
       // marks the end of a command
       commandReceived = true;
       return;
    }
    else if (incomingByte == '@')
    {
       // marks the start of a new command
       command = "";
       commandReceived = false;
       return;
    }
    else
    {
      command += incomingByte;
      return;
    }
    
  }
}

void handleCommand() {
  
  if (!commandReceived) return; // no command to handle
  
  // variables to hold the command id and the command data
  int id;
  int data[9];
  
  // NOT SURE WHAT TO DO HERE!!
  
  // flag that we've handled the command 
  commandReceived = false;
}

Say my PC sends the Arduino the string "@20,2000,5!". My sketch ends up with a String variable (called command) that contains "20,2000,5" and the commandRecieved boolean variable is set to True so the handleCommand() function is called.

What I would like to do in the (currently useless) handleCommand() function is assign 20 to a variable called id and 2000 and 5 to an array of integers called data, i.e: data[0] = 2000, data[1] = 5, etc.

I've read about strtok() and atoi() but frankly I just can't get my head around them and the concept of pointers. I'm sure my sketch could be optimised too.

Could anybody help out?

I've read about strtok() and atoi() but frankly I just can't get my head around them and the concept of pointers. I'm sure my sketch could be optimised too.

You won't until you try something.

The strtok() function takes a string to parse, and a set of delimiters. It returns a pointer to the portion of the string up to the first delimiter in the set.

char stuff[] = "20,2000,5";
char *token = strtok(stuff, ",");

So, print token, and see what it contains. Then, call atoi() with the token as input, and see what you get as output.

Note that to keep parsing the same string, the first argument needs to change to NULL.

int a = 0, b = 0, c = 0;
if(token)
  a = atoi(token);
token = strtok(NULL, ",");
if(token)
  b = atoi(token);
token = strtok(NULL, ",");
if(token)
  c = atoi(token);

[EDITED]

Thanks for the quick response!

Here’s my handleCommand() function:

void handleCommand() {
  
  if (!commandReceived) return; // no command to handle
  
  // variables to hold the command id and the command data
  int id;
  int data[2]; // up to three integers of data
  
  char cmd[command.length()+1];
  char *token = strtok(cmd, ",");
  
  // id
  if (token) {
    id = atoi(token);
    token = strtok(NULL, ",");
  }
  
  // data  
  for (int i = 0; i < 3; i++) {
    
    if (token) {
      data[i] = atoi(token);
      token = strtok(NULL, ",");
    }
    
  }
  
  // test
  Serial.println(command);
  Serial.println(cmd);
  Serial.println(id);
  Serial.println(data[0]);
  Serial.println(data[1]);
  
  // flag that we've handled the command 
  commandReceived = false;
}

The string I’m sending is:

@1,2,1!”

The output I’m getting is not correct though:

1,2,1

0
-9726
-9982

Something’s not quite right. Any ideas?

Fixed it!

Here’s the working function (I’m sure it could be improved though):

void handleCommand() {
  
  if (!commandReceived) return; // no command to handle
  
  // variables to hold the command id and the command data
  int id;
  int data[2]; // up to three integers of data
  
  char cmd[command.length()+1];
  command.toCharArray(cmd, command.length()+1);
  char *token = strtok(cmd, ",");
  
  // id
  if (token) {
    id = atoi(token);
    token = strtok(NULL, ",");
  }
  
  // data  
  for (int i = 0; i < 3; i++) {
    
    if (token) {
      data[i] = atoi(token);
      token = strtok(NULL, ",");
    }
    
  }
  
  // test
  Serial.println(command);
  Serial.print("id = ");
  Serial.println(id);
  Serial.print("data[0] = ");
  Serial.println(data[0]);
    Serial.print("data[1] = ");
  Serial.println(data[1]);
  
  // flag that we've handled the command 
  commandReceived = false;
}

Thanks again.

I was having exactly the same problems and I was just about to start a new thread when I saw this thread. As the problem seemed to be fixed here I simply edited your code to suit my data. However when I cam to verify I got this error

“expected primary-expression before “>” token”
“sketch_aug11b.cpp: In function ‘void handleCommand()’:
sketch_aug11b:44: error: expected primary-expression before ‘>’ token”

I’m using “<” & “>” to mark the beginning an end of the data instead of “@” & “!”

Here’s the code

String command; // a string to hold the incoming command
boolean commandReceived = false; // whether the command has been received in full

void setup() {
  // put your setup code here, to run once:
Serial.begin(115200);
}

void loop() {
  // main loop
  handleCommand();
}

void serialEvent(){
  while (Serial.available()) {
  // all we do is construct the incoming command to be handled in the main loop
    
    // get the incoming byte from the serial stream
    char incomingByte = (char)Serial.read();
    
    if (incomingByte == '>')
    {
       // marks the end of a command
       commandReceived = true;
       return;
    }
    else if (incomingByte == '<')
    {
       // marks the start of a new command
       command = "";
       commandReceived = false;
       return;
    }
    else
    {
      command += incomingByte;
      return;
    }
    
  }
}

void handleCommand() {
  
  if (>commandReceived) return; // no command to handle
  
  // variables to hold the command id and the command data
  
  int data[17]; // up to three integers of data
  
  char cmd[command.length()+1];
  command.toCharArray(cmd, command.length()+1);
  char *token = strtok(cmd, ",");
  
  
  // data  
  for (int i = 0; i < 17; i++) {
    
    if (token) {
      data[i] = atoi(token);
      token = strtok(NULL, ",");
    }
    
  }
  
  // test
  Serial.println(command);
  Serial.print("data[0] = ");
  Serial.println(data[0]);
  Serial.print("data[1] = ");
  Serial.println(data[1]);
  Serial.print("data[2] = ");
  Serial.println(data[2]);
  Serial.print("data[3] = ");
  Serial.println(data[3]);
  Serial.print("data[4] = ");
  Serial.println(data[4]);
  Serial.print("data[5] = ");
  Serial.println(data[5]);
  Serial.print("data[6] = ");
  Serial.println(data[6]);
  Serial.print("data[7] = ");
  Serial.println(data[7]);
  Serial.print("data[8] = ");
  Serial.println(data[8]);
  Serial.print("data[9] = ");
  Serial.println(data[9]);
  Serial.print("data[10] = ");
  Serial.println(data[10]);
  Serial.print("data[11] = ");
  Serial.println(data[11]);
  Serial.print("data[12] = ");
  Serial.println(data[12]);
  Serial.print("data[13] = ");
  Serial.println(data[13]);
  Serial.print("data[14] = ");
  Serial.println(data[14]);
  Serial.print("data[15] = ");
  Serial.println(data[15]);
  Serial.print("data[16] = ");
  Serial.println(data[16]);
  Serial.print("data[17] = ");
  Serial.println(data[17]);
  
  
  // flag that we've handled the command 
  commandReceived = false;
}

The error comes at the beginning of the
void handleCommand() {

on the

if (>commandReceived) return;

I’m sure I’ve missed something incredibly simple.

*EDIT

I think it might have just been a typo, I removed the “>” from if (>commandReceived) return; & it verified. I thought that line was a little odd. Sorry for the bother, im off to test and then ill come back and confirm.

Well that did seem to be the problem, although the data I'm getting through is not right at all, but I'm guessing that's a separate issue all together.

If anyone else has any comments, I'll be interested to hear them:)

I think you'll find it should be

  if (!commandReceived) return; // no command to handle

dxw00d: I think you'll find it should be

  if (!commandReceived) return; // no command to handle

If I do it this way IDE cannot verify as stated above, how strange. I posted the full code I'm using above, I don't suppose someone would try running it through their IDE and letting me know if it actually compiles.

This compiled ok.

String command; // a string to hold the incoming command
boolean commandReceived = false; // whether the command has been received in full

void setup() {
  // put your setup code here, to run once:
Serial.begin(115200);
}

void loop() {
  // main loop
  handleCommand();
}

void serialEvent(){
  while (Serial.available()) {
  // all we do is construct the incoming command to be handled in the main loop
    
    // get the incoming byte from the serial stream
    char incomingByte = (char)Serial.read();
    
    if (incomingByte == '>')
    {
       // marks the end of a command
       commandReceived = true;
       return;
    }
    else if (incomingByte == '<')
    {
       // marks the start of a new command
       command = "";
       commandReceived = false;
       return;
    }
    else
    {
      command += incomingByte;
      return;
    }
    
  }
}

void handleCommand() {
  
  if (!commandReceived) return; // no command to handle
  
  // variables to hold the command id and the command data
  
  int data[17]; // up to three integers of data
  
  char cmd[command.length()+1];
  command.toCharArray(cmd, command.length()+1);
  char *token = strtok(cmd, ",");
  
  
  // data  
  for (int i = 0; i < 17; i++) {
    
    if (token) {
      data[i] = atoi(token);
      token = strtok(NULL, ",");
    }
    
  }
  
  // test
  Serial.println(command);
  Serial.print("data[0] = ");
  Serial.println(data[0]);
  Serial.print("data[1] = ");
  Serial.println(data[1]);
  Serial.print("data[2] = ");
  Serial.println(data[2]);
  Serial.print("data[3] = ");
  Serial.println(data[3]);
  Serial.print("data[4] = ");
  Serial.println(data[4]);
  Serial.print("data[5] = ");
  Serial.println(data[5]);
  Serial.print("data[6] = ");
  Serial.println(data[6]);
  Serial.print("data[7] = ");
  Serial.println(data[7]);
  Serial.print("data[8] = ");
  Serial.println(data[8]);
  Serial.print("data[9] = ");
  Serial.println(data[9]);
  Serial.print("data[10] = ");
  Serial.println(data[10]);
  Serial.print("data[11] = ");
  Serial.println(data[11]);
  Serial.print("data[12] = ");
  Serial.println(data[12]);
  Serial.print("data[13] = ");
  Serial.println(data[13]);
  Serial.print("data[14] = ");
  Serial.println(data[14]);
  Serial.print("data[15] = ");
  Serial.println(data[15]);
  Serial.print("data[16] = ");
  Serial.println(data[16]);
  Serial.print("data[17] = ");
  Serial.println(data[17]);
  
  
  // flag that we've handled the command 
  commandReceived = false;
}

Yeah sorry, I'm a fool is what it comes down too.

In the original code "!" means the end of the data, but in the line

" if (!commandReceived) return;" ! must mean something else.

In my code the end of the data is represented by ">" so I exchanged all "!" for ">" & this broke the code :S

I changed it back to

" if (!commandReceived) return;" & my Data checks out.

One day I might understand something heh. Your help, much appreciated . Thanks.

http://arduino.cc/en/Reference/Boolean

The '!' means 'not', so "if (!commandReceived) return;" means if not (commandReceived is true) then return.

Ahh now that explains everything.

Thanks my friend

  while (Serial.available()) {
  // all we do is construct the incoming command to be handled in the main loop
    
    // get the incoming byte from the serial stream
    char incomingByte = (char)Serial.read();
    
    if (incomingByte == '>')
    {
       // marks the end of a command
       commandReceived = true;
       return;
    }
    else if (incomingByte == '<')
    {
       // marks the start of a new command
       command = "";
       commandReceived = false;
       return;
    }
    else
    {
      command += incomingByte;
      return;
    }

Regardless of what single character is actually received, return after processing that character. Do so in a while loop. Why? The while loop is silly, given that nothing happens if there is no serial data, and the function processes exactly one character if there is.

The only block that should contain a return statement is the one that deals with the ‘>’.

Hey there, could you elaborate as far as what you mean by the while loop being silly?

Forgive my noobness :slight_smile:

My understanding of the code goes like this…

while data is available via serial
place data in “incomingbyte”

if “incomingbyte” = “>” complete Boolean
and return for more data

else if “incomingbyte” = “<” start buffer “command”, Boolean incomplete
so return for more data

else place “incomingbyte” into buffer “command”, Boolean incomplete
so return for more data

This, at least to me, seems like a logical way of collecting the data until you have the completed command. The While loop itself says only to try to collect the data if there is serial data to collect. which also seems logical.

As I say I’m new to programing in general & I’m also teaching myself, so I’m prone to misunderstanding even the simplest of concepts, but equally open to any knowledge you can spare. So if you could clarify your statement it would be appreciated.
Thanks.

return; exits from the current function, returning to where it was originally called.

So you have a while loop that intends to process all the available data in the serial buffer, but for every character you read, you are exiting the function, which also exits the while loop, which completes defeats the intent of the while loop.

You should only return; if you have a complete command string. In all other cases, let the while loop process the next character, if it's available().

Ah now that makes sense, there's no point in breaking out of the while loop before you need to deal with the data, as the "while loop" loops anyway, heh!.

I've mainly been dealing with If statements & with out using Booleans, so I feel I've learnt a lot.

Much appreciated. I edited my code (removed the unneeded return; commands) and it works great.