Here is a sample problem sketch, the object of which is to split a string into sections and where the section contains subsections, to split those as well
For what it is worth this is running on an ESP32, hence the use of Serial.printf()
char arrayA[] = {"a b/c/d e f/g"};
void setup()
{
Serial.begin(115200);
delay(500);
Serial.printf("parsing %s\n", arrayA);
char * main = strtok(arrayA, " ");
while (main != NULL) //iterate throught the array
{
Serial.printf("main %s\n", main);
if (strchr(main, '/')) //if there is a / in the current string
{
subSplit(main); //split the sub string
}
main = strtok(NULL, " "); //where is strtok() continuing from ?
}
}
void loop()
{
}
void subSplit(char * sub)
{
char * subPart = strtok(sub, "/");
while (subPart != NULL)
{
Serial.print("subPart ");
Serial.println(subPart);
subPart = strtok(NULL, "/");
}
}
Here is the output from it
parsing a b/c/d e f/g
main a
main b/c/d
subPart b
subPart c
subPart d
As you can see, output stops after the first call to sunSplit(). As noted in the comments I suspect that the problem is that strtok() in setup() continues from where subSplit() left off rather from where it was previously in the array
If that is the case what do I make a copy of before calling subSplit(), else what is the problem and how do I solve it
My aplogies to any pointers that may have been damaged by the sketch
strtok is not thread safe, because it modifies the source, you are running 2 strtoks on the same source, hmm what could go wrong I wonder. I suggest you leave your obsession with strtok and switch to sscanf
You undoubtedly know this, but strtok() can do the parsing in one step, if you give it multiple delimiters. Of course the logic will be different in dealing with the results.
char * main = strtok(arrayA, " /");
Otherwise, just copy the separate substrings to new char arrays, after the first step, then process those substrings.
Understand that strtok does NOT create any new strings. Each time you call it, it replaces the delimiter character with a null, and returns a pointer to the beginning of the found string, WITHIN the original un-parsed string. You have to treat the returned string carefully to avoid clobbering, over-writing, or over-running, the original string. Unless you are VERY careful, do not interleave strtok calls that are dealing with different strings. You can look at the strtok source to see why these limitations exist.
You can do basically what you're trying to do, EXCEPT, in the first-level parse, you need to save the pointers returned by strtok, so you retain knowledge of where all the substrings start. Complete the first-level parse first, then use the saved pointers to do the second-level parse.
Or, after each call to strtok(), copy the returned string to a new string.
He provided all that was needed to re-create the issue.
Issues are often environment specific so I see no fault here.
Also, it isn't his fault that Arduino.cc has refused to add in a printf() method to the Print class for more than 10 years and many users over the years asking for it.
Pretty much every 3rd party Arduino core platform includes it.
Arduino.cc is the odd man out when it comes to this.
It is absolutely trivial to ADD printf to the Print class. Literally, about 30 seconds - a single cut and paste. They leave it out by default to save both FLASH and RAM.
Yes it is trivial to add, which is why it seems so odd that they have chosen not to add it.
I even updated an old Playground page more than 10 years ago to show how a printf() Print class method could be done. https://playground.arduino.cc/Main/Printf/
There is no "left out by default". It was consciously decided not to put it in at all.
What is also frustrating is that some people claim it would eat up resources.
But when not used it eats up no resources.
There is zero impact to FLASH and RAM usage if a printf() method is added to the Print class but is not used by the sketch code.
When it is there, only the users that choose to use it pay the approximately 1.5k of code space for it (on the AVR) and then depending on the implementation some stack space.
But there is a way to implement it on the AVR with out the added stack buffer.
The biggest veto to adding a printf() method to the Print class came from Tom Igoe himself.
He claimed that it was just too confusing for Arduino users.
Yet every 3rd party core has chosen to implement it.
And given printf() and its formatting strings has been around now over 50 years and is even implemented in many different languages, documentation on how it works is quite easy to find.
Well, that was an interesting experience and I have been put in my place, so here are some comments from me following replies received
I am not obsessed with strtok() but it does seem like the perfect tool for the job. Can sscanf() actually do what I need and split a string of arbitrary length into sections ?
This is a possible solution, but I dismissed it because the logic of dealing with the results sounds complicated, especially in the actual sketch that I encountered the problem with where the situation is more complicated
Ironically it is 2 stage processing that I am trying to get away from
You are right, I don't like it but not being thread safe does not appear to be the cause of the problem in this case, but what is the thread safe alternative ?
Don't forget that the statistics include posts made in my role as a moderator and are, therefore, inflated
When I first read this I had steam coming out of my ears, but on reflection you are right and I have noted your comment. See post #11
As I said it is 2 stage processing that I am trying to get away from
And the alternative is ?
I assume that in your screenshot that you are trying to point out that I did not use strtok_r() in the subSplit() function but that does not matter in this case because it is only the top level parsing that needs to be re-entrant. If, however, there were 3 (or more) levels then each level above the bottom would need to be re-entrant
Oh, and by the way, you are a naughty boy for posting a screenshot of code. With your post count you ought to know better. Touché
No, it is more suitable for parsing strings of known format….thinking about it, yes, you can skip sections of the string so yes you can have arbitrary length as long as there is a pattern to format
and as the code of the two functions really looks alike, you could condense it a bit further using 1 level of recursion and a flag to say if you are looking for tokens or items
char messageToParse[] = "a b/c/d e f/g";
const char* tokenSeparators = " ";
const char* itemSeparators = "/";
void handleItem(char* item) {
Serial.print("\t\t➜");
Serial.println(item);
}
void split(char* message, bool itemLevel = false) {
char * tokenPtr = nullptr;
char * startPtr = message;
if (itemLevel) Serial.write('\t');
Serial.print("parsing "); Serial.println(message);
do {
tokenPtr = strpbrk(startPtr, itemLevel ? itemSeparators : tokenSeparators);
if (tokenPtr != nullptr) {
char tmpChar = *tokenPtr;
*tokenPtr = '\0';
if (itemLevel) handleItem(startPtr); else split(startPtr, true);
*tokenPtr = tmpChar; // restore the string
startPtr = tokenPtr + 1; // get ready for the next token
} else if (itemLevel) handleItem(startPtr); else split(startPtr, true);
} while (tokenPtr != nullptr);
}
void setup() {
Serial.begin(115200); Serial.println();
split(messageToParse);
}
void loop() {}
That's nice because it even appears to handle empty tokens as found say in a .csv file.
I did a quick test with an NMEA sentence in the Woki simulator which @Koepel is popularising here:
well you would be better off checking if item is the nullptr in the handleItem() function (if you want to do more than printing it out) but the code does send an empty cString
