Serial.read() -> Multi-dimension Array -> analyze data -> format output

Projects Programming
1

Hi everyone,

I'm working on an Arduino project and am stuck. I've tried lots of things, in last 2-weeks and can't figure it out. Any help would be greatly appreciated!

Basically, our working script (on Arduino Uno) sends AT+ commands to an A7 GSM Board, and reports back the data. Here's the actual output from Serial.write(); :-

//--- Actual ouput is this -----

AT+CGDCONT=?
+CGDCONT: (1..7), (IP,IPV6,PPP),(0..3),(0..4)

OK

AT+CRLP?
+CME ERROR:58

AT+CGMI
Ai Thinker Co.LTD

OK

I need the output to be like this:

|---   Ok   ---| [AT+CGDCONT=? ] [+CGDCONT] (1..7), (IP,IPV6,PPP),(0..3),(0..4) 
|--- ERROR  ---| [AT+CRLP?     ] [+CME    ] ERROR:58
|---   Ok   ---| [AT+CGMI      ] [        ] Ai Thinker Co.LTD

From what I have been researching the last week, is that I think this needs to be put into a "multi-dimensial" ARRAY foreach "AT+" result, and then interrogate the ARRAY. So a typical i believe the ARRAY dataset would look like this??:

   //--- Assuming this is ARRAY Contents for "AT+CGDCONT" from above -----

   GSM_Array[0](i) = 'AT+CGDCONT'                                    //--- AT_COMMAND
   GSM_Array[1](i) = '+CGDCONT: (1..7), (IP,IPV6,PPP),(0..3),(0..4)' //--- AT_RESPONSE :1
   GSM_Array[2](i) = ' '                                             //--- AT_RESPONSE :2
   GSM_Array[3](i) = OK                                              //--- AT_RESPONSE :3

If the ARRAY worked, then, I assume we could do something like this, to process the ROW_3 of the Raw output...

    String AT_CMEError = "+CME ERROR:58";
    String AT_MsgError;
    String AT_CPlus = "+C";
    String AT_MsgCmd;
    
   //---- Check for: '+CME ERROR:' --------

   if (strcmp(GSM_Array[1](i),AT_CMEError)  == 0) {  //--Check if +CME ERROR occurs
      AT_MsgError = "|--- ERROR  ---|" ;
    }  else {
      AT_MsgError = "|---   Ok   ---|";
   }

Normally, I would use 'AWK/GREP/SED' in Korne or Bourne for this kinda thing. But this is C+
on Arduino. Nearest best thing I can think of was the following : (i know this is not good but i'm trying to figure it ):

      //---- Check for: "+C" MSG detail --------
      //---- Exclude  : +CME ERROR:58    -------- 

      if (strcmp(GSM_Array[1](i),AT_CPlus)  == 0) && (strcmp(GSM_Array[1](i),AT_CMEError)  != 0) 
      { 
       AT_MsgCmd="+C????";  //--- eg. +CGDCONT,  how to "AWK & CUT" the GSM_Array[1](i) ??? 
                            //--- Would this be a 'chomp'???
      } else {
       AT_MsgCmd="";        //-- This is for RESPONSE like "AT+CGMI" it has only 2 rows of data returned.
      fi

Then finally I need to print the report. Also need to "RPAD" Column 2 & 3.. I've used this kind of approach before, but do not know how it translates to ARRAY data:

    char myBuffer[35] = {0};
    sprintf(myBuffer, "%-6s|", "|AT+CGMI");     //-- This prints:  |AT+CGMI     |
    Serial.print(myBuffer);

I guess the Print out would be something along the lines of this? means some kinda rpad technique to be inserted:

        Serial.print(AT_MsgError);
        Serial.print(" [");
        Serial.print(<RPAD>(GSM_Array[0](i),6);    //rpad 6x spaces---how to do this?
        Serial.print("] [");
        Serial.print(<RPAD>(AT_MsgCmd,6);          //rpad 6x spaces---how to do this?
        Serial.print("] ");
        Serial.println(<RPAD>(GSM_Array[1](i),45); //rpad 45x spaces---how to do this? 
        delay(500);

Thanks again for you help.
Really do appreciate it.

kr.

[ATTACHMENT:] Here's the Working script, RAW output, and the Required Format we need.....

AT_cmds_Test.V.1.1.4_FORMAT_REPORT_.txt.ino (1.15 KB)

AT_cmds_Test.V.1.1.4_RAW_OUTPUT_.txt.ino (555 Bytes)

AT_cmds_Test.V.1.1.5.ino (2.15 KB)

2

OK, the RPAD problem is the easy one...

On big computers with lots of memory, it's easy to just make big strings bigger. On the Arduino, this would end up tying up lots of memory and cause lots of data to be moved around in memory. You need to think a little more like a microcontroller. In a case where you have limited memory, you only want to store the bare minimum. If you have to format something nicely for the output then do it on the output, not in memory.

So right-padding a variable length string to N characters long, just send out the appropriate number of spaces on the output, followed by the string. Don't ever store those N characters.

void rpad(char* S, int N) {
  int charCount = strlen(S);
  while(charCount < N) {
    Serial.write(' ');
    charCount++;
  }
}

This would be used like:

  rpad(GSM_Array[0](i),6);
  Serial.print(GSM_Array[0](i));

Your input parsing problem is a little more difficult. The method you propose assumes that you have the entire strings in memory first. Then you can do strcmp() on the strings to see what you have. But that leaves a big problem: how do you know when you have the entire string? Related to this: how do you know when a string is starting?

Since you can't change the AT commands that the thing is sending to you, you need to do a bit more work than Robin2's excellent Serial Input Basics. When I'm faced with this kind of thing, I end up with a big state machine. It starts with the input buffer empty. It looks at the first character that arrives on Serial. If that matches the first character of any of the strings we're looking for, then store it. Now there's probably only 2 or 3 target strings that have that first char. Look at the next one when it arrives. Maybe that leaves only 2 that it could be. When you get down to only one possibility, then keep storing characters until you reach the known end of that particular type of command. Now you know what you have and you can parse out the details from that command. You also know that the next command is coming next, so reset the buffer to empty and start from the top.

3

GSM_Array[0](i) = looks like a call to a function.
That shouldn't be on left side of an assignment.

4

Thanks for the quick replies!

@MorgranS - thanks for explaing the logic of how the buffering works. I'm so used to bigMEM, it's nice to work in confined space again. Also the RPAD was great, I just need to work on some Char conversion errors before I can implement. My main focus is getting the ARRAY to be delim'd and formatting the output. So, i followed your breadcrumb to @RobinS and found Reply #72. I did a bit of rework to handle CHAR instead of INT and increased the numChars[index] size and it works ok...... but.

Enter AT+INFO:

<AT+CPBR=?` +CPBR: (1-250),20,20  OK >
<<<|AT+CPBR=?| +CPBR: (1-250),20,20  OK |>>>

It seems strtok() doesn't seem to detect "Control Characters". If it was working I would expect the result to be:

<<<|AT+CPBR=?| +CPBR: (1-250),20,20 | OK |>>>

I reset the GSM to switch ECHO=off and use CHAR (0x04)(EOS) as the output delimiter. AT+DST command only lets us use ASCII chars 1...31, all non-printable. Here's the RAW output(copied to notepad) and what it looks like on the screen (2nd):

//--- In notepad -----//

<AT+CPBR=?`
+CPBR: (1-250),20,20

OK
>

//--- As seen in terminal (edited out the CR)------//

<AT+CGMI` Ai Thinker Co.LTD OK >
<AT+CGMR` V03.03.20161229019H03 OK >
<AT+CGSN` 867567021266240 OK >
<AT+CGDCONT=?`  +CGDCONT: (1..7), (IP,IPV6,PPP),(0..3),(0..4) OK >

I tried to implement isControl() to detect the 0x04 but it did not work. However, when I copy the 0x04 from RAW output and run the sketch from (https://www.arduino.cc/en/Tutorial/CharacterAnalysis) it tells me that it is indeed a "Control" character.

Also, tried the "strncmp()" suggestion from @Robin Reply #74. No joy also.

Is there something basic that I am missing here?

ps. void parseData() is where i tried the delim. work.

/***
Author     : Robin2@arduino.cc
Source     : https://forum.arduino.cc/index.php?topic=288234.msg2459466#msg2459466
**/
// Receive with start- and end-markers combined with parsing

const byte numChars = 64;
const byte numCharsRSP = 255;
char receivedChars[numCharsRSP];
char tempChars[numCharsRSP];   // temporary array for use by strtok() function

// variables to hold the parsed data
char msgFromPC_CMD[numChars] = {0};
char msgFromPC_RSP1[numCharsRSP] = {0};
char msgFromPC_RSP2[numCharsRSP] = {0};

boolean newData = false;

 
//============

void setup() {
    Serial.begin(57600); //9600);
    Serial.println();
    Serial.println("Enter AT+INFO:");
    Serial.println();
}

//============
void loop() {
    recvWithStartEndMarkers();
    if (newData == true) {
        strcpy(tempChars, receivedChars);
        parseData();
        showParsedData();
        newData = false;
    }
}
//============
void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;
 
    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();
        Serial.print(rc);
        //Serial.print("");
        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
//                if (ndx >= numChars) {
//                    ndx = numChars - 1;
//                }
                if (ndx >= numCharsRSP) {
                    ndx = numCharsRSP - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                //  receivedChars[ndx] = '^';
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }
        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

void parseData() {
      // split the data into its parts
    char * strtokIndx; // this is used by strtok() as an index
      //----- AT CMD -----------------------//
      strtokIndx = strtok(tempChars,"`");        // get the first part - the string
      strcpy(msgFromPC_CMD, strtokIndx);         // copy it to msgFromPC_CMD
      
      //----- AT RSP(1) --------------------//                
      strtokIndx = strtok(NULL, 0x04);            // this continues where the previous call left off
      
//      if (isControl(strtokIndx)){
//        Serial.print("isControl");
//        strcpy(msgFromPC_RSP1, strtokIndx);
//      }

//      if (strncmp(strtokIndx, "0x", 2) = 0) {
//          strcpy(msgFromPC_RSP1, strtokIndx);
//      }

      strcpy(msgFromPC_RSP1, strtokIndx);
      
      //----- AT RSP(2) --------------------//
      //strtokIndx = strtok(NULL, "`");
      strtokIndx = strtok(NULL, 0x04);
      strcpy(msgFromPC_RSP2, strtokIndx); 
}

//============
void showParsedData() {
      
      Serial.println("");
      Serial.print("<<<");
      Serial.print("|");
      Serial.print(msgFromPC_CMD);
      Serial.print("|");
      Serial.print(msgFromPC_RSP1);
      Serial.print("|");
      Serial.print(msgFromPC_RSP2);
      Serial.println(">>>");
      
}
5 
      strtokIndx = strtok(NULL, 0x04);            // this continues where the previous call left off

So you're searching the string previously given for the characters that are in memory location 4?

      strtokIndx = strtok(NULL, "\x04");            // expecting to find 0x04 as the next delimiter
6

Hi @MorganS, thanks for the update. I'm not familiar with the syntax of "\x04". I did do some searching and the results show mostly buffer related code manipulation, is this on track?

I did try it but it did not work for me. I ran the standard AT+ responses through encoder ASCII text,Hex,Binary,Decimal,Base64 converter to determine the HEX values of the output. Then used strtok(NULL, 0x0a) but it seems not to detect the 0x0a as a delimiter. But it does detect ok strtok(NULL, ":") .... I dont nderstand
why this is so.

/********************
 * AT+ Response DECODE
 * ******************
-----------------------------------------------------------------------------
<<<|AT+CGMR| V03.03.20161229019H03  OK ||>>>
-----------------------------------------------------------------------------
<                       | 3C 0A 
AT+CGMR                 | 41 54 2B 43 47 4D 52 0A
                        | 0A
V03.03.20161229019H03   | 56 30 33 2E 30 33 2E 32 30 31 36 31 32 32 39 30 31 39 48 30 33 0A 
OK                      | 0A 4F 4B 0A 
>                       | 3E
-------------------------------------------------------------------------------
<<<|AT+CPAS| +CPAS|0  OK |>>>
-------------------------------------------------------------------------------
<                       | 3C 0A 
AT+CPAS                 | 41 54 2B 43 50 41 53 0A 
+CPAS:0                 | 0A 2B 43 50 41 53 3A 30 0A 
                        | 0A 
OK                      | 4F 4B 0A 
>                       | 3E
-----------------------------------------------------------------------------
Result:   <<<|AT+CGDCONT=?| +CGDCONT| (1..7), (IP,IPV6,PPP),(0..3),(0..4) |>>>
-----------------------------------------------------------------------------
<                       | 3C 0A 
AT+CGDCONT=?            | 41 54 2B 43 47 44 43 4F 4E 54 3D 3F 0A 
                        | 0A 
+CGDCONT: (1..7), (IP,IPV6,PPP),(0..3),(0..4) | 2B 43 47 44 43 4F 4E 54 3A 20 28 31 2E 2E 37 29 2C 20 28 49 50 2C 49 50 56 36 2C 50 50 50 29 2C 28 30 2E 2E 33 29 2C 28 30 2E 2E 34 29 0A
>                       | 3E
-----------------------------------------------------------------------------
Result: <<<|AT+CMER=?| +CMER| (3),(0),(0),(0,2)  OK |>>>
-----------------------------------------------------------------------------
<                       | 3C 0A 
AT+CMER=?               | 41 54 2B 43 4D 45 52 3D 3F 0A 
+CMER: (3),(0),(0),(0,2)| 0A 2B 43 4D 45 52 3A 20 28 33 29 2C 28 30 29 2C 28 30 29 2C 28 30 2C 32 29 0A 
                        | 0A 
OK                      | 4F 4B 0A 
>                       | 3E
***/

Is it possible(and simple/better) to convert the rc = Serial.read(); from CHAR to STRING as each one is processed/buffer and interpolate them that way?

Or, can the whole sequence of buffered chars from rc = Serial.read(); be populated into one single Variable/Array/String, then do some work on the whole thing, before printing to serial?

I ran it at 9600 BAUD to test if there was a timing issue, but same result.

Penny for your thoughts?

  // split the data into its parts
  char * strtokIndx; // this is used by strtok() as an index
  //----- AT CMD -----------------------//
  strtokIndx = strtok(tempChars,  " ");   //"`");        // get the first part - the string
  strcpy(msgFromPC_CMD, strtokIndx);         // copy it to msgFromPC_CMD
  //----- AT RSP(1) --------------------//                
  strtokIndx = strtok(NULL, ":");   //"\x04");            // this continues where the previous call left off
  strcpy(msgFromPC_RSP1, strtokIndx);
  //----- AT RSP(2) --------------------//
  strtokIndx = strtok(NULL, "\x04"); //0x0a); //" ");  //0x10);  //"\x4");  //"0x0a");  //":");  //"\x4");  //"\0x3a");
  strcpy(msgFromPC_RSP2, strtokIndx);
  //----- AT RSP(3) --------------------//
  strtokIndx = strtok(NULL, 0x0a);  //" ");  //0x20);  //"\x4");   //" ");  //"\x20" );  //"\x4");   //"\0x4a");
  strcpy(msgFromPC_RSP3, strtokIndx);
7

Full script for above:

AT_Cmd_Parser.v1.3.ino (5.38 KB)

8

"\x04" is a string. You can tell that because of the double quotes.

The backslash () is the escape character. It says that what comes next is a code for a special character that can't be typed on the keyboard. You usually see this used to create newlines with the escape sequence \n

The x says that we want to give the character value in hexadecimal. So the next two digits are the hex code.

The reason for all of this is the strtok() function only works on strings. It does not deal with single characters unless you give it a string that is one character long.

9

Is there a character you can depend on not getting in a response? If there is you could replace non-printable characters with something strtok() can handle when reading the input. For example:

receivedChars[ndx] = (rc <= 0x1f) ? '~' : rc;

Also, strtok() gives you pointers into tempChars and adds null terminators for you, so you should not need to copy the parts of the input into separate buffers.

10

Thanks again @MorganS, that really helped me to disambiguate it clearly.

@BlueEyes, good idea on the "exclusion" approach. The range for the CR & LF strings may be set to any CHAR between DEC[ 1...31 ] with the a7(GSM) chip. This is done via,

ATS3=10  -- set CR = 0x0a
ATS4=13  -- set LF = 0x0d

So I reset them to these values and did some unit testing for both 0x0a & 0x0d with the snippet that you supplied. Still seems to not be performing as anticipated. Did I get the placement incorrect in the IF Statement???

I output both the value of rc & receivedChars[ndx] to Serial.print() to see what was happening. Nothing else was changed in the code from the most recent attachment.

EDIT: actually I changed parseData(); to handle the '~' delim.

parseData(){
.......
strtokIndx = strtok(tempChars,"~");

.......
}

Thanks again....

.......
if (rc != endMarker) {
    receivedChars[ndx] = (rc <= 0x1f) ? '~' : rc;  //--- Blue Eyes 
    // (condition) ? (if_true) : (if_false);       //--- shorthand
    //----DEBUG----//
    Serial.print(rc);
    Serial.print(receivedChars[ndx]);
    //------------//
    //receivedChars[ndx] = rc;
    ndx++;
    if (ndx >= numCharsRSP) {
        ndx = numCharsRSP - 1;
    }
......

The Unit Test Output:

ROW 1: value set for (rc <= CHAR) ? '~' : rc;
ROW 2: Serial.print() of both (rc) & receivedChars[ndx]
ROW 3: "void showParsedData();"

(rc = 0x1f)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>

(rc > 0x1f)

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>

(rc < 0x1f)

  AATT++CCGGMMRR    ++AATT++CCGGMMRR::VV0033..0033..2200116611222299001199HH0033    OOKK  
<<<| AT+CGMR  +AT+CGMR|V03.03.20161229019H03  OK ||>>>

(rc <= 0x1f)

  AATT++CCGGMMRR    ++AATT++CCGGMMRR::VV0033..0033..2200116611222299001199HH0033    OOKK  
<<<| AT+CGMR  +AT+CGMR|V03.03.20161229019H03  OK ||>>>

(rc <= "0x1f")

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>

(rc <= "\0x1f")

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>

(rc <= "\x1f")

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>


(rc <= 0x0d)

  AATT++CCGGMMRR    ++AATT++CCGGMMRR::VV0033..0033..2200116611222299001199HH0033    OOKK  
<<<| AT+CGMR  +AT+CGMR|V03.03.20161229019H03  OK ||>>>


(rc <= "0x0d")  

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>

(rc <= "\0x0d") 

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>

 (rc <= "\x0d")

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>


(rc = 0x0d) 

  AATT++CCGGMMRR    ++AATT++CCGGMMRR::VV0033..0033..2200116611222299001199HH0033    OOKK  
<<<| AT+CGMR  +AT+CGMR|V03.03.20161229019H03  OK ||>>>

(rc > 0x0d)

 ~A~T~+~C~G~M~R~ ~ ~+~A~T~+~C~G~M~R~:~V~0~3~.~0~3~.~2~0~1~6~1~2~2~9~0~1~9~H~0~3~ ~ ~O~K~ ~
<<<|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|||>>>


(rc <= 0x1d)

  AATT++CCGGMMRR    ++AATT++CCGGMMRR::VV0033..0033..2200116611222299001199HH0033    OOKK  
<<<| AT+CGMR  +AT+CGMR|V03.03.20161229019H03  OK ||>>>