Loosing Serial data

Hi Everyone.
I’m having some issues with loosing serial data that I hope some one can help me.
I’m using a device that converts TCP to Serial (Wifly).
Running at 19200 baud, everything works wondeful, but at 38400 I can’t process the data fast enough without dropping data.
Here is the function I’m using to interpret data:

void pushbuffer(byte inStr)
{
      memcpy(&m_pushback[0], &m_pushback[1], 31); 
      m_pushback[30]=inStr;
      m_pushback[31]=0;
      if (strncmp("GET /", m_pushback, 5)==0) reqtype=1;
      if (strncmp("GET /wifi", m_pushback, 9)==0) reqtype=2;
      if (strncmp("GET /r", m_pushback, 6)==0) reqtype=3;
      if (strncmp(authStr, m_pushback, 12)==0) auth=true;
}

And inside loop(), I have this:

      if (Serial.available()>0) pushbuffer(Serial.read());

I guess with all the memcpy and strncmp going on, the processor takes too long and the serial buffer overflows?
Is that what’s going on?
Any ideas on how to optimize my function?

Thank you
RI

Just out of curiosity, why don't you fill the buffer from the front?

That way, you could skip some of the strncmp() calls, if the length of the string being checked is less than the length of the string being compared to.

Then. order the strncmp() statements in order by length. If you have determined that the string starts with "GET /wifi", there is no reason to check that the string starts with "GET /r" or "GET /" or authStr. So, add some elses in front of the subsequent ifs.

Hi Paul,

I don't think I can use strncmp if I start feeding from the front. The first letter to come in is G. When E comes in, G would get pushed back and the string would turn to "EG" and so forth. It would be comparing: "ifiw/ TEG" to "GET /wifi" The length of the string is really long. In fact, here it is:

GET / HTTP/1.1 Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, / Accept-Language: en-us User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) Accept-Encoding: gzip, deflate If-Modified-Since: Sun, 18 Sep 2005 00:11:06 GMT If-None-Match: "246bf76e5bbc51:df7" Host: 10.1.10.44 Connection: Keep-Alive Authorization: Basic c2F0OmF1dG8=

What I really need to find out out of this very long string is: 1) Does it contain "GET / " 2) Does it contain "GET /r" 3) Does it contain "GET /wifi" 4) Does it contain "c2F0OmF1dG8=" The method I'm using can easily find out the first 3 because the Serial buffer is 128 characters and I'll always have the first 128 bytes available to compare. But, since 4th comparison is at the end of this long variable, if I increase above 19200baud, I start loosing data and the comparison fails.

I don't think I can use strncmp if I start feeding from the front. The first letter to come in is G. When E comes in, G would get pushed back and the string would turn to "EG" and so forth.

The whole idea of appending the next character read is to prevent having to shuffle everything every time a character is read.

int index = 0; The first character, G, gets put in the position defined by index, and the index gets incremented (to 1). The next character, E, gets put in the position defined by index, and the index gets incremented (to 2). The next character, T, gets put in the position defined by index, and the index gets incremented (to 3).

Hey Paul,

When index get to 31, I would still need to shift the whole thing anyway, wouldn't it?

When index get to 31, I would still need to shift the whole thing anyway, wouldn't it?

As opposed to index getting to zero?

As opposed to index getting to zero?

I did not understand your question :(

What happens in your code when there is not room for another character? You can make the array larger, but the process gets even slower.

In my method, you can make the array larger, and there is no performance penalty.

I'm still unsure of what happens when the buffer gets filled. Lets say you received 31 characters and m_pushback = "GET /wifi\n\rAccept: image/gif, i" What do you do with the next character "m" that comes in? Wouldn't you still have to push all the data to the left and make m_pushback = "ET /wifi\n\rAccept: image/gif, im"?

You have a fixed size array for the incoming data. When the array is full, it's full. Shifting everything one character to the left means that character is lost. Not shifting everything means that the incoming character is lost.

It is up to you to size the array appropriately, and so something with the data in the array before the array gets full.

Oh, maybe my code was misinterpreted. memcpy(&m_pushback[0], &m_pushback[1], 31); This actually shifts the first character out of the array to make room for the new chararcter. So, let's supposed the at the beginning, I have m_pushback filled with 0s. The first character comes in, memcpy is called and mpushback becomes " G" The second character comes is, memcpy is called and mpushback becomes " GE" All the way to the 31st character and mpushback becomes "GET /wifi\n\rAccept: image/gif, i" The next character will call memcpy and mpushback becomes "ET /wifi\n\rAccept: image/gif, im" And it keeps pushing everything to the left for every character that comes in. The array is fixed, but I just keep shifting the array to the left so I can use strncmp to find out if I have the piece of string I'm looking for or not. Was it better explained?

I understood what you were doing from the first post.

Filling the array in the other direction is faster, and achieves the same results.

When the array contains "GET /wifi", you need to do one things, regardless of whatever else the unread serial data may contain. The same is true when the array contains "GET /r". "GET /" will be trickier, since that will appear in the array before "GET /r" or "GET /wifi", so perhaps you need to make sure that you have read 9 characters before doing any strncmp's.

Hey Paul,

My problem is not the beginning of the string. It's actually the end of the string. So, using your method, what would I do when index=31 and mpushback="GET /wifi\n\rAccept: image/gif, i"? Where am I going to place the next character, which is "m"?

Make the array bigger. There is nothing magic about 32. Make it 128.

No, there is nothing magic about 32, but I think if I go 128, I may be using more RAM than I have available. Besides, I can't grow the array to the size of the incoming string. Let me make it simpler. Forget about the comparisons at the beginning of the string. They will always be available due to the serial internal buffer. What I really need to find out and I'm loosing data thus can't compare is: How would I find out if the incoming string contains "c2F0OmF1dG8="?

At 38K baud your Arduino still has time to do thousands of instructions to do between each characters (baud is not char/second, and for this serial line is more like bits/second).

As you write "convert to serial", I presume you do some Serial.print() calls, and these DO block you program, so much so that maybe your push routine doesnt get called.

The internal serial buffer (the one you pick from when you call Serial.read) is 64 bytes long IIRC, so you can ignore Serial.available for that long. If you wait longer it does not matter if your character buffer is 32 or 128 bytes long.

The above "facts" mesh with you observation that you miss something when upping the baud rate.

Now, the long and tedious discussion about whether to fill the array from the back or the beginning ... never mind. I'll just note that I had not thought of doing it that way round (your way) before nor anywhere else. But you are wasting CPU cycles with needless shuffling around. (oops - there I said it anyhow :-[ ;))

How would I find out if the incoming string contains "c2F0OmF1dG8="?

That string specifically? Simply don't store anything until you receive a c. Then, if the next character is not a 2, don't save it, and set index back to 0. When you have received a c and the next character is a 2, save all the characters up to the =. Then, see if what is saved is that string. If you run out of room in the array, it was the wrong c2 that started the string.

Ok, I changed my code to go the other way around, like others have suggested. Here is what I got:

void pushbuffer(byte inStr)
{

      m_pushback[m_pushbackindex]=inStr;
      if (reqtype>0)
      {
            if (authStr[m_pushbackindex]==inStr) m_pushbackindex++; else m_pushbackindex=0;
            if (authStr[m_pushbackindex]==0) auth=true;
      }
      else
      {
            m_pushbackindex++;
            if (m_pushbackindex==32) m_pushbackindex=0;
            if (strncmp("GET / ", m_pushback, 5)==0) reqtype=1;
            if (strncmp("GET /wifi", m_pushback, 9)==0) reqtype=2;
            if (strncmp("GET /r", m_pushback, 6)==0) reqtype=3;
      }
}

Thanks for the tips :) Only thing is I found out that the above code was also dropping data. So, after a little troubleshooting on my code I found out that the delays from the onewire library were too long and I think were causing the serial buffer to fill up. So, I just created a lot of if conditions to go check for serial data when I'm handling the onewire data.

Thanks a lot for your help!!

RI