Serial data parsing with strtok() giving some issues

Projects Programming
1

Context:

A small ATTINY1607 based project with an RTC, which can receive, over serial, the time data and set the time to RTC.
The uC wakes up on a button press and shows the set time and then goes to sleep. While awake it can receive the above mentioned serial data.
Additionally, I'm trying to send another value with the current time data (details below), which will determine, in next cycle for how long the uC should be awake for.
The serial data is coming from a webpage (using web-serial)

Problem explanation:

Note: before moving forward, I would like to state that, I will focus on parts of code base without overwhelming anyone. If any more parts from the code base is required, please state, I will try to be provide, if it exists.

The data sent from the js side is in this structure:

HH:MM:SS:WEEKDAY:DD:MM:YYYY:STAY_AWAKE_TIME.

For e.g: 02:18:19:6:25:06:2021:5

Main part on JS Side (using web-serial):

syncbtn.addEventListener("click", () => {
    // -- Get the time
    const now = new Date;
    
    // Here the vaklue of "Delay" stands for how long the uC should be stay awake for. 
    // To set it dynamically, it is added to end of the rest of serial data set.
    // ** Delay value comes from a HTML select object 
    var delay_selection = document.getElementById("delays");
    var delay_in_ms = Number(delay_selection.value);

    serialData = now.getHours()+":"+
                        now.getMinutes()+":"+
                        now.getSeconds()+":"+
                        now.getDay()+":"+
                        now.getDate()+":"+
                        now.getMonth()+":"+
                        now.getFullYear()+":"+
                        delay_in_ms;

    // -- write the Serial Data
    if(port !=null){
        writeToStream(serialData);
    }else{
        console.log("Not writing to Serial Port as it wasn't created/selected!");
    }
});

function writeToStream(...lines) {
    const writer = outputStream.getWriter();
    lines.forEach(line => {
      console.log("[SEND]", line);
      writer.write(line + "\n");
    });
    writer.releaseLock();
}

Now in the Parser side of the uC, it looks for a '\n' as the end of data stream and up until then it stores everything in a char array. Once end of stream is reached, it starts parsing the char array data structure with the help of the delimators and assigns the respectiove values to the variables to be used further (for example: setting time and uC's wakeup period).

Main part on Arduino Side:

void parseDataArray() {
  if (newDataArrived) {
    newDataArrived = false;
    totalDelimators = 0;

    // Count how many delimators (in our case that is ':' of byte value 10) are there
    for (int i = 0; i < int(sizeof(dataArray)); i++) {
        if (dataArray[i] == ':') {
            totalDelimators++;
        }
    }

    // Datastructure:
    // 02:18:19:6:25:06:2021    (totalDelimators == 6)
    // 02:18:19:6:25:06:2021:5  (totalDelimators == 7)

    // Check received data's format & integrity
    if (totalDelimators >= 6) {  // or 6/7 based on the stream ends with year value or with additional delay value
        char * strtokIndx; // this is used by strtok() as an index
        strtokIndx = strtok(dataArray, ":"); // get the first part - the string

        hrToBeSet = atoi(strtokIndx);        // convert this part to an integer
        strtokIndx = strtok(NULL, ":");      // this continues where the previous call left off
        minToBeSet = atoi(strtokIndx);       // convert this part to an integer
        strtokIndx = strtok(NULL, ":");
        secToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        weekdayToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        dateToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        monthToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        yearToBeSet = atoi(strtokIndx);
    
        // [TBD / WIP bc not converting to int properly ]
        strtokIndx = strtok(NULL, ":");
        new_stayAwakeFor = (atoi(strtokIndx))*1000;    // where x is in sec which needs to be converted in milli seconds.Hence *1000

        setNewTime = true;
    }
  }
}

As you might have seen in the code part, I have highlighted, the last bit from the data structure (STAY_AWAKE_FOR) is creating the issue.
I'm not sure if it os JS side issue in the way it creates teh string or the uC's parsing method issue!

Things I have tried:

When in JS side, I remove the last bit of data and only send time data, i.e: when the data structure is HH:MM:SS:WEEKDAY:DD:MM:YYYY and on the Arduino side, I parse only upto the ...:YYYY part of the data structure, it works flawlessly.

JS side:

.
.
.

    // serialData = now.getHours()+":"+
    //                     now.getMinutes()+":"+
    //                     now.getSeconds()+":"+
    //                     now.getDay()+":"+
    //                     now.getDate()+":"+
    //                     now.getMonth()+":"+
    //                     now.getFullYear()+":"+
    //                     delay_in_ms;

    serialData = now.getHours()+":"+
                        now.getMinutes()+":"+
                        now.getSeconds()+":"+
                        now.getDay()+":"+
                        now.getDate()+":"+
                        now.getMonth()+":"+
                        now.getFullYear();
.
.
.

uC side:

.
.
.

// Check received data's format & integrity
    if (totalDelimators == 6) {  // or 6/7 based on the stream ends with year value or with additional delay value
        char * strtokIndx; // this is used by strtok() as an index
        strtokIndx = strtok(dataArray, ":"); // get the first part - the string

        hrToBeSet = atoi(strtokIndx);        // convert this part to an integer
        strtokIndx = strtok(NULL, ":");      // this continues where the previous call left off
        minToBeSet = atoi(strtokIndx);       // convert this part to an integer
        strtokIndx = strtok(NULL, ":");
        secToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        weekdayToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        dateToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        monthToBeSet = atoi(strtokIndx);
        strtokIndx = strtok(NULL, ":");
        yearToBeSet = atoi(strtokIndx);

        setNewTime = true;
    }

.
.
.

The parsing doesn't work when the last bit ( ...:STAY_AWAKE_TIME ) is added to the serial string!.
No idea what I'm doing wrong here!. Definitely missing something.
Any help would help :)
Thanks in Advance for guidance!

2

I would resolve that first. Write a short sketch to receive the data on the serial port and mirror it to the serial monitor. Is the data what you expect?

Should there be a delimiter after STAY_AWAKE_TIME ?

3

This does full Serial Console commands and its actually fairly easy in this example.

in the loop
call this Method


String getSerialCommand() {
  String retCmd = "";
  if (Serial.available())
    retCmd = Serial.readString();
  return retCmd;
}

// for example
String serialCmd = getSerialCommand();

// use these string utils also in the example above

// splits String After the delim
String splitAt(String d1, String delim) {
  int pos = d1.indexOf(delim) + delim.length();
  if (pos < -1)
    return "";
  return d1.substring(pos);
}
// splits String Before the delim
String splitBefore(String d1, String delim) {
  int pos = d1.indexOf(delim);
  if (pos < -1)
    return "";
  return d1.substring(0, pos);
}
// split string is a bit more complicated in a dynamic and hidden way but this is the work around
// make a global variable for the substrings to go into
String split_return[255];
// use this method to return the amount of substrings in a int and put the return in split_return
int split(String str, String delim){
  
  int stringCount=0;
  
  while (str.length() > 0)  {
    int index = str.indexOf(delim);
    if (index == -1) {
      split_return[stringCount++] = str;
      break;
    } else {
      split_return[stringCount++] = str.substring(0, index);
      str = str.substring(index+1);
    }
  }

  return stringCount-1;
    
}

Please let me know if I am being too confusing?

4

malloc

memory allocate
https://en.cppreference.com/w/c/memory/malloc

strtok is fine and all perhaps a ref to dataArray each time its called using a substr and a string library.

5

In your example, you try to read the 7th Param inside the 6 param if clause…

7

It's >= 6 and not ==6 but I have also tried with 7 earlier.

8

Okay found it!
My dataArray size was not changed/modified to accomodate the last bit in the data structure. :no_mouth:

Explanation:
char dataArray[21]; for incoming data structure: HH:MM:SS:W:DD:MM:YYYY
and when I appended that with another byte i.e. STAY_AWAKE_TIME (on the js side like so HH:MM:SS:W:DD:MM:YYYY:STAY_AWAKE_TIME), I forgot to change the char dataArray[21]; to char dataArray[23]; on the uC side of things (even though I changed the number of delimiters)

And BTW, there is already a delimiter for the STAY_AWAKE_TIME, just before it and there should not be anything after.

9

It was the size of the dataArray which wasn't large enough for the data-structure.
I had to change that and it worked.

//char dataArray[21]; // HH:MM:SS:W:DD:MM:YYYY
char dataArray[23];   // HH:MM:SS:W:DD:MM:YYYY:STAY_AWAKE_FOR

I'm now thinking how to have a dynamic array (pointers + malloc() may be ) ...

May be increase the size from 23 to something a bit bigger for if I want to add few more things to the data structure, later down the lane.

int sizeOfDataStructure = int(sizeof(char) * 23);
char *dataArray = malloc(sizeOfDataStructure);

.
.
.
// Then after parsing and assigning with strtok() etc. , if I should do:
free(dataArray);

Everything seems to compile except I get a warning:

... warning: invalid conversion from 'void*' to 'char*' [-fpermissive] char *dataArray = malloc(sizeOfDataStructure);

Something to do with how a malloc() is defined in the stdlib?
How to rectify and avoid

Also a side note, using pointers here consumes more storage by about approximately 5% and dynamic memory by 1%

So is it a good idea anyways or I should just make the size larger in char dataArray[<size>]; to something big to accomodate addition of new data to the data-structure?