Searching for words on serial without saving input data?

Projects Programming
1

When reading serial sometimes the input data is too large to save, but we need to search inside it anyway , for a certain words .

Right now we accumulate all the data and only then look for the specific word. I would like to do that while reading, without saving the data , so to improve this one :

  boolean waitFor(char  *target, long timeout)
        {

                unsigned long tm=millis();
                  while( 1 )
                  {
                    if( millis() - tm >= timeout ) 
                      return 0;

                    char wifiContent[50]={0};
                    int readWifiIndex=0;

                    while( wifiSerial.available()  ) 
                    {  

                        if(readWifiIndex<50)
                        {                 
                             wifiContent[readWifiIndex]=(char)wifiSerial.read();
                             readWifiIndex++;
                             delay(1);  
                        } 


                    }


                //**check of specific words 
                    if( strstr(wifiContent,target) )
                       return 1;


                  }

        }

Update: I have changed my code to search WHILE i read, and everything is great now .

2

I have changed my code to search WHILE i read, and everything is great now .

I would be interested to see how you have done it.

3

UKHeliBob:
I would be interested to see how you have done it.

The KMP algorithm can do it (in O(n)); link: Knuth–Morris–Pratt algorithm - Wikipedia

kind regards,

Jos

4

JosAH:
The KMP algorithm can do it (in O(n)); link: Knuth–Morris–Pratt algorithm - Wikipedia

kind regards,

Jos

Nice, and with explanation / documentation for a change. Thanks for the link.

But it does not satisfy the "spec" / requirement - not to store data received but process them "on the fly".
But I admit I did not read the whole article in details, it may do it.

I have a small task to check for duplicate entries in a given ( pretty short , about 30 characters) string, so I will give this a try.

Since my "motion detector" is essentially comparing old (picture) frame with new one I am sure KMP algorithm will be useful.

5

Vaclav:
But it does not satisfy the "spec" / requirement - not to store data received but process them "on the fly".

Yes it does; from the very first paragraph from that link:

In computer science, the Knuth–Morris–Pratt string searching algorithm (or KMP algorithm) searches for occurrences of a "word" W within a main "text string" S by employing the observation that when a mismatch occurs, the word itself embodies sufficient information to determine where the next match could begin, thus bypassing re-examination of previously matched characters.

(bold face is mine).

kind regards,

Jo

6

Does the KMP algorithm lend itself to searching for one of several words as required by the OP, although the number of words to be searched for has not been specified.,

7

UKHeliBob:
Does the KMP algorithm lend itself to searching for one of several words as required by the OP, although the number of words to be searched for has not been specified.,

Yep, the KMP algorithm can find several words in a text in one pass (it basically is a state machine), but in reply #7 the OP started talking about 'duplicate entries' in a text and that complicates matter a lot ...

kind regards,

Jos

8

MorganS:
Hint: the KMP algorithm is not a serious suggestion for an Arduino. But it's fun to think about it.

I disagree - the KMP algorithm is ideally suited for this. It is very space efficient, very fast and is easy to implement.

9

Its actually very easy and I have 2 ways to do it. one looks char by char and the other looks at the whole word.

Char by Char

template <typename T, size_t N>
byte FindStr(T(&str)[N], char buf)
{
  static int i = 0;
  if (*(str + i) == buf)
  {
    i++;
    if ( i == (N - 1))
      return true;
  }
  else
  {
    i = 0;
    return false;
  }
}

void setup()
{
  // put your setup code here, to run once:
  Serial.begin(115200);
}

void loop()
{
  // put your main code here, to run repeatedly:
  if (Serial.available())
  {
    if (FindStr("Test", Serial.read()))
      Serial.println(F("String was found"));
    else
      Serial.println(F("String was not found"));
  }
}

Whole word (uses a buffer)

char StringIn[20];
int found;
char charData;
int i = 0;
int flag;
byte led = 13;


#include<string.h>
void setup()
{
  pinMode(led, OUTPUT);
  Serial.begin(115200);
}

void loop()
{
  if (Serial.available())
  {
    FindStr("Test", Serial.read());
  }
}

byte FindStr(char * str, char charData)
{
  if (charData == '.' || charData == ' ' || charData == ',' || charData == '\n')
  {
    found = strcmp(str, StringIn);
    if (found == 0) 
    {
      Serial.println("got it");
      flag = 0;
    }
    else
    {
      if (!flag)
      {
        Serial.println("Scanning");
        flag = 1;
      }
    }
    memset(StringIn, 0, sizeof(StringIn));
    i = 0;
  }
  else
  {
    StringIn[i] = charData;
    StringIn[i + 1] = '\0';
    i++;
  }
  delay(20);
}
10

HazardsMind:
Its actually very easy and I have 2 ways to do it. one looks char by char and the other looks at the whole word.

Char by Char

template <typename T, size_t N>

byte FindStr(T(&str)[N], char buf)
{
  static int i = 0;
  if (*(str + i) == buf)
  {
    i++;
    if ( i == (N - 1))
      return true;
  }
  else
  {
    i = 0;
    return false;
  }
}

void setup()
{
  // put your setup code here, to run once:
  Serial.begin(115200);
}

void loop()
{
  // put your main code here, to run repeatedly:
  if (Serial.available())
  {
    if (FindStr("Test", Serial.read()))
      Serial.println(F("String was found"));
    else
      Serial.println(F("String was not found"));
  }
}





Whole word (uses a buffer)

char StringIn[20];
int found;
char charData;
int i = 0;
int flag;
byte led = 13;

#include<string.h>
void setup()
{
  pinMode(led, OUTPUT);
  Serial.begin(115200);
}

void loop()
{
  if (Serial.available())
  {
    FindStr("Test", Serial.read());
  }
}

byte FindStr(char * str, char charData)
{
  if (charData == '.' || charData == ' ' || charData == ',' || charData == '\n')
  {
    found = strcmp(str, StringIn);
    if (found == 0)
    {
      Serial.println("got it");

print str here ??
and return ??

flag = 0;
    }
    else
    {
      if (!flag)
      {
        Serial.println("Scanning");
        flag = 1;
      }
    }
    memset(StringIn, 0, sizeof(StringIn));
    i = 0;
  }
  else
  {
    StringIn[i] = charData;
    StringIn[i + 1] = '\0';
    i++;
  }
  delay(20);
}

Did I missed something?

Where is return value (byte) set in second definition of FindString set?

Can you demonstrate using either method that the raw incoming data is not being collected and than processed?

I have tried that but I feel using the input from COM monitor is not the way to test this code, since it gets offered to Serial after "enter" key all at once.

The code probably needs some tracking Serial to demonstrate that, but it is too late in soggy Houston now.

11

BenStlr:
There are 2 options , you are here to help , you are not here .

There's also the option for the OP to help others to help him/her.
Like, for instance, not posting code snippets.

Or the option for the OP to foxtrot oscar over to avrfreaks.

Amongst the options are counted ....

12

Did I missed something?

Where is return value (byte) set in second definition of FindString set?

No, I copied the name of the first one, because the second one was originally called "DidYourStringComeIn"

Can you demonstrate using either method that the raw incoming data is not being collected and than processed?

If you use the serial monitor then yes, it will be collected then processed. The only way that it might work is if you change the serial monitor buffer size to 1 instead of 64, but then i'm not sure what will happen to the rest of the message.

13

I have tried that but I feel using the input from COM monitor is not the way to test this code, since it gets offered to Serial after "enter" key all at once.

So, do what the grown-ups do, and use a proper terminal emulator.

14

BenStlr:
When reading serial sometimes the input data is too large to save, but we need to search inside it anyway , for a certain words .

Right now we accumulate all the data and only then look for the specific word. I would like to do that while reading, without saving the data , so to improve this one :

  boolean waitFor(char  *target, long timeout)

{

unsigned long tm=millis();
                  while( 1 )
                  {
                    if( millis() - tm >= timeout )
                      return 0;

char wifiContent[50]={0};
                    int readWifiIndex=0;

while( wifiSerial.available()  )
                    {

if(readWifiIndex<50)
                        {               
                            wifiContent[readWifiIndex]=(char)wifiSerial.read();
                            readWifiIndex++;
                            delay(1); 
                        }

}

//**check of specific words
                    if( strstr(wifiContent,target) )
                      return 1;

}

}

You are right about reading without buffering. I mention that here from time to time, I've been doing just that since the mid 80's on PC's and here and there on Arduino.
Maybe I should finish that sequence-matching class I started but it's hard to give up what I used to get paid for. It does not try to match just one sequence. it does not Block Execution.

One thing you should not do though is to make this into a blocking function.

In my signature space below are 3 addresses to Nick Gammon tutorials. The second one contains a good start on the use of a simple state machine to read data without blocking. The first explains the concept of non-blocking very simply and clearly.

Would you like to be able to read serial data as you have described while blinking a status led, watching other inputs and making other outputs all "at the same time"? Yes? Those first two tutorial blogs will give you the keys to drive that code.

Vaclav is being a bit sharp and bitter. But true, Robin has a preferred way which is true of most experienced programmers. Robin prefers to buffer and use string.h commands whereas I left that path decades ago. Why waste time between the arrival of serial data when you can parse and lex on the fly?

15

Groove:
So, do what the grown-ups do, and use a proper terminal emulator.

I like PuTTY but for some things Hyperterminal is good.