Help with Processing a Char Array Populated with Serial Data

Hello,

I would be grateful of some help with the code I am working on below.

//  Using the Least Number of Libraries as Possible to Save Space / Resources
#include <SoftwareSerial.h>
SoftwareSerial MYGSM(2,3);      // Assign GSM Software Serial to Digital Pins 2 & 3 [RX/TX]

// SMS Packet Markers & Indicators 
#define SOP '<'                 // Start of Packet Marker
#define EOP '>'                 // End of Packet Marker
#define SEP ';'                 // Separation Marker for Indicating Separation Between Multiple Commands

//  All Commands Have a Three Letter Prefix Followed Directly by Their Values.  
//  Example Command Packet Format:-
//  Multiple Commands: <MCT1;txF145.000;TAE0>
//  Single Command: <MCT1> 


// Flags
bool started = false;           // Start Reading Serial Data (SMS)
bool ended = false;             // Finish Reading Serial Data (SMS)

// Data Variables
char smsMsg[128];               // Array of Char Buffer for Holding SMS Message
byte index;                     // Position Keeping Index Count for SMS Message Buffer
char *search;                   // Used for Searching the SMS Message Array Buffer

char mobNumber[] = "+441234567890";

//============

void setup() {
    Serial.begin(9600);         // Hardware Serial to Serial Monitor Begin at 9600 Baud
    MYGSM.begin(9600);          // GSM Software Serial Begin at 9600 Baud     
}

//============

void loop()
{
    rxSMS();                   // Get the SMS Message
      
}

// FUNCTION TO REQUEST AND GET THE SMS MESSAGE FROM THE SIM800L GSM MODULE
void rxSMS(){
    
    // Handshake Type Wake Up & Do Stuff Command...
    if(MYGSM.available() <= 0){
      
        // Set the GSM Module's SMS mode to ASCII
        MYGSM.println(F("AT+CMGF=1\r\n"));
        delay(100);
   
        // Set the SMS Notification Mode on the GSM Module - Some Modes PUSH the Message / Notification Straight Out Through Serial - We Don't Want This!
        MYGSM.println(F("AT+CNMI=2,1,0,0,0\r\n"));
        delay(100);

        // Request SMS Message from SIM Memory Storage Location #1 (Messages are deleted after use so new msgs should always be in location 1)
        MYGSM.println(F("AT+CMGR=1\r\n"));
        delay(100);   // Delays Add a Small Amount of Time for the Instructions to be Processed & Responded to.
    }
    
    // IF There's Data at the Serial Port, Start Reading...
    if(MYGSM.available() > 0){
        index = 0;     
        while(MYGSM.available() > 0){
            char inChar = MYGSM.read();    // Read each character into the reusable inChar variable

            //----------------------------------------------------------------------------------------------------------------------------------------------
            //  CODE SHOULD GO HERE FOR FIRST READING THE SMS DATA INTO A CHAR ARRAY BUFFER,, THEN DO ERROR CHECKING OF THE DATA, THEN VALIDATE THE
            //  SENDER'S PHONE NUMBER AND FINALLY, IF ALL CONDITIONS ARE MET, EXTRACT THE PACKET DATA CONTAINING THE COMMAND(S) FROM THE SMS & PROCESS THEM
            //-----------------------------------------------------------------------------------------------------------------------------------------------
            //
            //1. READ ALL OF SMS INTO CHAR ARRAY (BUFFER) INC HEADER INFO SUCH AS TIME STAMP, SENDER NUMBER ETC.
            //2. CHECK SMS FOR ERRORS
            //3. EXTRACT AND VALIDATE SENDER'S NUMBER FROM SMS
            //4. PROCESS SMS TO FIND PACKET DATA (COMMANDS)
            //
            // NOTE... SMS MESSAGES APPEAR TO COME IN THE FORM OF HEADER DATA FIRST AND THEN A NEW LINE AND OR CARRIAGE RETURN FOLLWED BY THE MESSAGE DATA (PACKET).....
            // SCAN FOR NEW LINE OR CARRIAGE RETURN TO DETECT THE NEXT LINE BEING THE MESSAGE CONTENT OR PACKET!

            // Store the Received Char Data in to the smsMsg Char Array Buffer
            if(index < 127){
                smsMsg[index] = inChar;
                index++;
                smsMsg[index] = '\0';  
            }
              
 
           
        } // End WHILE GSM Available
    
    } // End IF GSM Available

    // HAND CONTROL OVER TO PROCESSING
    processSMS();


    delay(5000);              // Wait Some Time - Meant to Simulate the Periodic Passage of Time While the Atmega is in Low Power Sleep Mode - [FOR TESTING ONLY] 

} // End rxSMS Function


// VALIDATE SENDER FUNCTION - EXTRACTS THE SMS SENDER'S NUMBER FROM THE SMS HEADER AND VALIDATES IT
void validateSender(){
  
}


// SORT, PROCESS AND DO STUFF WITH THE COMMANDS RECEIVED VIA THE SMS "PACKET"
void processSMS(){

   Serial.println(F("SMS MSG CONTENT:-"));
   int i=0;
    while(i<90){
        Serial.print(smsMsg[i]);
        i++;
        
    }
    search = strstr(smsMsg, "+441234567890");
    if( search != NULL){
        Serial.println("");
        Serial.print(F("Found String at SMS Message Buffer Index "));
        int pos = search - smsMsg;
        Serial.print(pos);
        Serial.println("");
        Serial.print(F("Result: "));
        
        //Print Out the Result From the Location Found...
        int x = pos;
        while(x < pos+13){
            Serial.print(smsMsg[x]);
        x++;
    }
    }else{
        Serial.println(F("Not Found!"));
    }

    // LOOK AT STRTOK and ATOI!

    if (strcmp(smsMsg,mobNumber)==0){
      Serial.println(F("Match Verified"));

    }
   // Serial.println("");
   // Serial.println("");

    
}

I am trying to store serial data into an array of char so I can then process it and look for specific strings.

So far I’ve got the program to read the serial data and store it in an array of char called smsMsg.

From there I can print out the contents of the array (SMS Message) and I can also find and print out the senders Mobile number.

The Problems I’m having:-

The first problem I’m having is that my smsMsg array is also storing the OK responses from the GSM module but I can’t figure out how to either ignore them before writing the serial data to the array or remove them from the array after writing.

The next problem I’m having is that I want to compare some values from one char array with another but I’m not sure how. What I am trying to do is compare a Master Mobile Phone Number stored in a char array called mobileNumber with the Sender’s Mobile Phone Number stored in among data in the smsMsg array.

Then I can validate the sender of the incoming SMS with a stored master number.

Thanks.

The first problem I’m having is that my smsMsg array is also storing the OK responses from the GSM module but I can’t figure out how to either ignore them before writing the serial data to the array or remove them from the array after writing.

Serial data transmission is meant to be completely asynchronous. That is, you send some data. You have NO expectations that any response will arrive OR that it will be in response to what you sent.

You are trying to force serial data transmission (sending AT commands IS serial data transmission) to fit a synchronous mold. You have a LOT of work to do to make that work.

“the serial data” includes ALL data coming from the device. “before writing the serial data to the array” doesn’t make sense.

    if(MYGSM.available() <= 0){

There can NEVER be a negative number of bytes that have not been read.

The next problem I’m having is that I want to compare some values from one char array with another but I’m not sure how.

Mr. Google does. The function strcmp() could prove useful.

Hi Paul S and thanks for replying. I was rather hoping to get a reply from you. Some parts of the code I’ve used is based on an example of yours I found in reply to another old thread on these forums - though I suspect you’ve probably already spotted and recognised your work in my code posted above?

Looking through your example, playing around with it and understanding how it works has helped a great deal so thank you!

PaulS:
Serial data transmission is meant to be completely asynchronous. That is, you send some data. You have NO expectations that any response will arrive OR that it will be in response to what you sent.

You are trying to force serial data transmission (sending AT commands IS serial data transmission) to fit a synchronous mold. You have a LOT of work to do to make that work.

“the serial data” includes ALL data coming from the device. “before writing the serial data to the array” doesn’t make sense.

I’ve read enough of your replies around this forum to know, understand and appreciate that you are trying to make me think about the problem rather than give the typical spoon feeding… but man, at this time of the morning that just blew my mind lol…

I’m think I’m going to have to take some time and let what you’ve said process and sink in and then get back to you on that one :slight_smile: hehe.

PaulS:

    if(MYGSM.available() <= 0){

There can NEVER be a negative number of bytes that have not been read.

Sorry, my bad. I was playing around with various checks and conditions at that point (while, If etc.) and must have overlooked the “<” because it was working. I will amend that. Thanks for pointing that out!

PaulS:
Mr. Google does. The function strcmp() could prove useful.

As silly as it sounds I keep forgetting / overlooking the fact that an Array of Char is a string and I can use functions like strcmp(). I think my determination to avoid the String object is causing me to confuse myself over the use of lower case s string?

So I’ve been pretty much looking at and treating my smsMsg Char Array just like any other array - if that makes sense? So with that, I would have gone ahead and processed the contents of the Char Array using loops and index position keeping variables and would have done the comparisons char by char which sounds to me now like I would have been re-inventing the wheel and making life much harder for myself?

As mentioned, as difficult as it is proving to re-wrap my head around, I am deliberately going out of my way to use Char Arrays instead of String Objects because of what I’ve read about how they can cause fragmentation problems and that they can eat up dynamic memory.

And with that, it’s back to Google to read some more…

Thanks again!

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data. There is also a parse example.

...R

PaulS:
Serial data transmission is meant to be completely asynchronous. That is, you send some data. You have NO expectations that any response will arrive OR that it will be in response to what you sent.

You are trying to force serial data transmission (sending AT commands IS serial data transmission) to fit a synchronous mold. You have a LOT of work to do to make that work.

“the serial data” includes ALL data coming from the device. “before writing the serial data to the array” doesn’t make sense.

So I’ve been staring at your reply for a while now and going back and forth on Google and I think I understand what you’re driving at now…

When I perform the handshake with the GSM module, I issue it the following commands:-

AT+CMGF=1 followed by AT+CNMI=2,1,0,0,0 and THEN I issue AT+CMGR=1

But what I don’t do is read or clear the responses from the first two AT commands which are sat in the Serial Buffer along with the SMS message I requested with the final AT command…?

So of course when I do a serial read, I get the WHOLE lot back including the OK responses from the AT commands I’ve issued and didn’t clear.

At least I think that’s right. It seems to make sense now I’ve thought about it.

Think of sending serial data like posting a notice on a bulletin board. What happens when you do that? Nothing. There is NO immediate response. So, you move on.

Later on, you walk by the bulletin board again, and see 3 replies to your posting. Do you have any idea the order that they were posted? Do you have any idea when they were posted? Nope. Not a clue.

THAT is asynchronous communication.

Now, one of the responses could have been the owner of the bulletin board approving your post, by stamping it with the date and time it was noticed. Is that response important? No. It's like the AT command responding "OK".

It's unfortunate that AT command responses are not standardized, with a reply size, so you know how much data to read AND when the complete response has arrived, AND some indication of what the response is a response to. But, that is a fact of life, and no amount of wishful thinking is going to change that.

If you need to know which reply goes with which command, you have to wait for a reply, a complete reply, and nothing but a complete reply before you send another command.

Does that mean you simply sit on your butt waiting for a reply? No. That would be like sitting in front of the bulletin board waiting for a reply. "Hey, move it, we need to mow the grass here". Well, the Arduino has other things it could be doing.

I think you can see that forcing asynchronous communication into a synchronous mold (you give a command; the modem responds immediately) is not going to be easy. The AT command might take some time to execute. Getting the number of messages to be read is quick. Getting the nearest cell tower's ID may not. It may not respond at all. You'll need to deal with all of these cases.

Thank you PaulS and you too Robin2!

Your analogy makes perfect sense now, Paul.

One thing that sticks out with it particularly is that I've increased the Software Serial Buffer from 64 to 128 and wondered why my data was getting cut off.

I then realised that I wasn't allowing sufficient time between sending the AT command to get my SMS Message and the message to fill up the Serial Buffer. So I would get some data when the rxSMS function was first called and the rest of the data would be in the Serial buffer and read when the function was called a second time.

So to liken it to your well put analogy, I was trying to read the notice on the notice board before it had finished being written :)

So what I've done now is to send the first two AT commands (CMGF and CNMI) then I wait a short while for their responses to arrive in the Serial buffer

I then I do a while Serial.Available and Serial.read() to "clear" the OK messages from the Serial Buffer and THEN issue the final AT command to retrieve the SMS message.

BUT because there is nothing to be done while waiting for the SMS data to populate the Serial Buffer, I call a short delay after issuing the last AT command requesting the SMS message to allow the Serial Buffer to populate fully. Probably not the most efficient way since I guess the time delay can vary on many factors such as how much data is being read, how fast it's being read etc.

But now I don't have the unwanted OK messages stored at the beginning of the smsMsg Char Array but all the critical data is being read and stored as I'd hoped. :) :)

Thanks!!

Now that I am getting the results I am expecting and I am able to store the Serial Data that I do want in my smsMsg Char Array, I’ve begun to work on processing the Data a little.

The first process I’m working on involves validating the SMS Message Sender to ensure the person sending SMS commands is authorised to do so.

To achieve this I’ve arrived at the code below which checks the contents of smsMsg Char Array and compares it against a hard coded “Master” mobile number which is also stored in a Char Array called mobNumber.

I’ve used strstr() to do the comparison and it seems to work well.

If the contents of the mobNumber Char Array are found in the smsMsg Char Array then my program prints out a message on the Serial Monitor stating that a match has been found so the SMS Sender must be valid. If no match is found then the user is invalid.

//  Using the Least Number of Libraries as Possible to Save Space / Resources
#include <SoftwareSerial.h>     // Software Serial Library
SoftwareSerial MYGSM(2,3);      // Assign GSM Software Serial to Digital Pins 2 & 3 [RX/TX] - GSM USES HARDWARE SERIAL IN MAIN PROJECT!

// SMS Packet Markers & Indicators - NOT IMPLEMENTED YET
#define SOP '<'                 // Start of Packet Marker
#define EOP '>'                 // End of Packet Marker
#define SEP ';'                 // Separation Marker for Indicating Separation Between Multiple Commands

//  All Commands Have a Three Letter Prefix Followed Directly by Their Values.  
//  Example Command Packet Format:-
//  Multiple Commands: <MCT1;txF145.000;TAE0>
//  Single Command: <MCT1> 

// Flags - NOT IMPLEMENTED YET
bool started = false;                 // Start Reading Serial Data (SMS)
bool ended = false;                   // Finish Reading Serial Data (SMS)

// Data Variables
char smsMsg[128];                     // Array of Char Buffer for Holding SMS Message
byte index;                           // Position Keeping Index Count for SMS Message Buffer
char *search;                         // Used for Searching the SMS Message Array Buffer

char mobNumber[] = "+440123456789";   // MASTER MOBILE NUMBER - ONLY COMMANDS FROM THIS NUMBER ARE VALID

//============

void setup() {
    Serial.begin(9600);               // Hardware Serial to Serial Monitor Begin at 9600 Baud
    MYGSM.begin(9600);                // GSM Software Serial Begin at 9600 Baud     
}

//============

void loop()
{
    rxSMS();                          // Get the SMS Message
    delay(5000);                      // Wait Some Time - Simulates Low Power Sleep Mode - [FOR TESTING ONLY]  
}

// FUNCTION TO REQUEST AND GET THE SMS MESSAGE FROM THE SIM800L GSM MODULE
void rxSMS(){
    
    // Handshake Type Wake Up & Do Stuff Command...
    if(MYGSM.available() == 0){
      
        // Set the GSM Module's SMS mode to ASCII
        MYGSM.println(F("AT+CMGF=1\r"));
        delay(100);
   
        // Set the SMS Notification Mode on the GSM Module - Some Modes PUSH the Message / Notification Straight Out Through Serial - We Don't Want This!
        MYGSM.println(F("AT+CNMI=2,1,0,0,0\r"));
        delay(100);

        // "Purge" the Serial Buffer of "OK" Responses from the Last Two AT Commands Issued. We are Not Interested in Confirmation Responses at this Point in Time.
        serialPurge();

        // Request SMS Message from SIM Memory Storage Location #1 (Messages are deleted after use so new msgs should always be in location 1)
        MYGSM.println(F("AT+CMGR=1\r"));
        delay(250);   // !IMPORTANT - Add a Small Time Delay for the Serial Buffer [ increased to 256] to Populate Before Reading it. Without this, Data Gets Chopped at the End.
    }
    
    // IF There's Data at the Serial Port, Start Reading...
    if(MYGSM.available() > 0){
        index = 0;     
        while(MYGSM.available() > 0){
            char inChar = MYGSM.read();    // Read each character into the reusable inChar variable

            // Store the received Serial data up to 127 Chars, with the 128th Char being the NULL Terminator.
            if(index < 127){
                smsMsg[index] = inChar;   // Write Char to Index Position n
                index++;                  // Advance to the Next Index Position
                smsMsg[index] = '\0';     // Write NULL Terminator - Last Char Stored in the Array will Always be NULL
            }
              
           
        } // End WHILE GSM Available
    
    } // End IF GSM Available

    // HAND CONTROL OVER TO PROCESSING FROM HERE ON OUT....
    processSMS();
     
} // End rxSMS Function

// SORT, PROCESS AND DO STUFF WITH THE COMMANDS RECEIVED VIA THE SMS "PACKET"
void processSMS(){

    // WE JUST PRINT THE CONTENTS OF THE SMSMSG ARRAY HERE FOR NOW.........
    Serial.println(F("SMS MSG CONTENT:-"));
    int i=0;
    while(i<sizeof(smsMsg)){
        Serial.print(smsMsg[i]);
        i++;
        
    }  

    // HAND CONTROL OVER TO THE SENDER VALIDATION FUNCTION FOR A SHORT WHILE.....
    validateSender();

    // IF SENDER IS VALID, RETURN CONTROL BACK FROM VALIDATING TO PROCESSING FOR PROCESSING ANY / ALL SMS  COMMANDS
    
}

// VALIDATE SENDER FUNCTION - EXTRACTS THE SMS SENDER'S NUMBER FROM THE SMS HEADER AND VALIDATES IT
void validateSender(){

    // ALSO LOOK AT STRCMP, STRSTR, STRTOK and ATOI AS OTHER USEFUL FUNCTIONS FOR MANIPULATING THE DATA!

    // Find the SMS Senders Number from the Data Stored in the smsMsg Array
    search = strstr(smsMsg, mobNumber);

    // IF a Match / Result is Found, Sender is VALID
    if( search != NULL){
        Serial.println(F(""));
        Serial.print(F("SEARCH RESULT: "));
        Serial.print(F("Match Found at smsMsg Array Index ["));
        int pos = search - smsMsg;
        Serial.print(pos);
        Serial.print(F("] ("));
        
        //Loop through from the identified position index and print out the chars
        int x = pos;
        while(x < pos+13){    // Print out Only 13 Chars (the number of chars that make up the senders number) from the position identified
            Serial.print(smsMsg[x]);
        x++;
    }
    Serial.print(F(")"));     // SPACING FOR ASTHETICS
    }

    // NO Match / Result Found, Sender is INVALID - REJECT SMS
    else{
        Serial.println(F(""));
        Serial.print(F("SEARCH RESULT: "));
        Serial.print(F("No Match Found - INVALID SENDER!"));
    }
    Serial.println(F(""));    // TWO BLANK LINES FOR SERIAL PRINT ASTHETICS ONLY
    Serial.println(F(""));
  
}

// DELETE ALL SMS MESSAGES STORED ON THE SIM CARD FUNCTION - NOT ENTIRELY CONVINCED OF THIS FUNCTION'S EFFECTIVENESS IN ITS CURRENT FORM...
//void delSMS(){
    //if(messageEnd == true){
       // MYGSM.println("AT+CMGD=1,4\r\n");
        //messageEnd = false;
       // Serial.println(F("Messages Deleted")); // [SERIAL CONSOLE MESSAGE FOR TESTING PURPOSES ONLY]
    //}
//}

// QUICK AND EASY METHOD FOR GETTING RID OF UNWANTED DATA WAITING IN THE SERIAL BUFFER
void serialPurge(){
    while(MYGSM.available()> 0){
        MYGSM.read();
    }
    delay(10);    // Just in case.....
}

I know that strcmp() was recommended but is there any reason not to use the strstr() method I am currently using to compare the Mobile Numbers?

Thanks!

pi_and_chips: One thing that sticks out with it particularly is that I've increased the Software Serial Buffer from 64 to 128

Do you mean that you modified the SoftwareSerial library?

There is no need to do that. You could receive 1000 bytes with a 64 byte buffer - probably also with a 16 byte buffer.

The purpose of the buffer is to act as a temporary store for incoming data until your program gets around to emptying the buffer. If you read the buffer frequently the buffer does not need to be big.

...R

I know that strcmp() was recommended but is there any reason not to use the strstr() method I am currently using to compare the Mobile Numbers?

strstr() and strcmp() do different things. strstr() searches a string for a string. strcmp() compares two strings.

If you have an string, "I love ice cream", and want to determine if the string contains "chocolate", then comparing the two strings using strcmp() clearly doesn't make sense. It should also be obvious that if you want to know if the first string contains "ice" that that is not the same is wanting to know if the first string IS "ice". strstr() will tell you whether or not the string contains "ice" or "chocolate" or whatever. strcmp() won't.

But, if you know that the smsMsg string contains a phone number, starting at position 6, and ending before position 17, and you extract that portion to another array, msgCameFrom, you can then use strcmp() to determine if msgCameFrom contains the same information as the mobNumber array.

Thank you both!

@Robin2

I originally increased the buffer size because the serial data was getting cut short. I will reduce it back to 64 and see what happens and take it from there. If there's no need to meddle with libraries, I would prefer not to.

@PaulS

Yes, sorry, that's my bad, I keep getting my terminologies mixed up.

So what I've done now is to first find if the mobile number sub-string is present in the smsMsg Array and if it is, I use strncpy to copy all the characters of the sub-string across into a fixed size temporary char array called tempMobNumber.

From there, I use strcmp to compare the hard coded "Master" mobNumber against the tempMobNumber. If the comparison == 0 then I know that the two numbers match so therefore the SMS sender is Valid.

Hopefully I've got it right now :)

I could have done it the way you suggested, Paul, by copying the characters from a known start and end index position but the extra search I perform before the comparison ensures I know the exact index position for what it is I am looking for as the basis for my copy and compare later.

While I believe the Mobile Number should start from the same index position in the smsMsg array every time, I'm not confident enough to put money on it lol. That little search I perform before comparing should hopefully guarantee I am indeed copying and comparing what I expect to be. If that makes sense?

So making the assumption that I've got everything with the SMS Sender validation in order, I can say that wasn't actually as bad as I thought it was going to be once I'd got my head around things a bit better.

It's even jogged my useless memory to some extent. :)

And now comes the interesting bit of my program.... Extracting and processing the commands...

I was dreading this bit but now, I am quite looking forward to getting stuck in and having a go. I feel like my "writers block" has lifted somewhat and like I've been re-acquainted with an old forgotten toolbox full of once favored tools....

Thanks again! I'm off to take my new found confidence and make that keyboard smoke.. hehe

:)