Parsing AT Commands via Serial Returns Results out of Order

Hello all!

I am attempting to communicate with a SIM7600 G-H using the Software Serial library on an Arduino Uno (ELEGOO UNO R3 to be precise).
I am able to get responses from the modem but the responses don't really make sense when I get to the really important stuff (hitting a public API I'm using for testing). The incoming message is all jumbled up and out of place compared to where I expect it...Please see my code snippet below as well as a response snippet.

#include <Arduino.h>
#include "SoftwareSerial.h"
#include <stdio.h>
#include <string.h>

#define rxp 2
#define txp 3

SoftwareSerial mySerial = SoftwareSerial (rxp,txp); 

const unsigned int MAX_MESSAGE_LENGTH = 400;
String data;

void mySerialRead() {
  while (!mySerial.available()) 
  {
    //wait for data
  }
   //Check to see if anything is available in the serial receive buffer
 while (mySerial.available() > 0)
 {
  //Read the next available byte in the serial receive buffer
  char inByte = mySerial.read();

  //Message coming in (check not terminating character) and guard for over message size
  static char input_line [MAX_MESSAGE_LENGTH];
  static unsigned int input_pos = 0;

  switch (inByte)
    {
    case '\n':   // end of text
      input_line [input_pos] = 0;  // terminating null byte
    
      // terminator reached! process input_line here ...
      Serial.println(input_line);
      // reset buffer for next time
      input_pos = 0;  
      break;

    case '\r':   // discard carriage return
      break;

    default:
      // keep adding if not full ... allow for terminating null byte
      if (input_pos < (MAX_MESSAGE_LENGTH - 1))
        input_line [input_pos++] = inByte;
      break;
    }
  }
}

void apiTest() { 
  String http_str = "AT+HTTPPARA=\"URL\",\"http://worldtimeapi.org/api/timezone/America/Chicago\"\r";//hitting this API just to see if I can get a chunk of data > buffer size
  
  Serial.println("init");
  mySerial.print("AT+HTTPINIT\r"); //starts http service.
  delay(200); 
  mySerialRead();
  Serial.println("END INIT");

  delay(2000);

  Serial.println("http string");
  mySerial.print(http_str); //Set URL parameter
  delay(200);
  mySerialRead();
  Serial.println("END HTTP STRING");

  delay(2000);

  Serial.println("http action");
  mySerial.print("AT+HTTPACTION=0\r"); //0 = get, 1 = post
  delay(200);
  mySerialRead();
  Serial.println("END HTTP ACTION");

  delay(5000);

  Serial.println("read http data");
  mySerial.print("AT+HTTPREAD=0,350\r"); //I expect a message of 351 bytes so I chose 350 here for maximum
  delay(200);
  mySerialRead();
  Serial.println("END READ");

  Serial.println("term");
  delay(500);
  mySerial.print("AT+HTTPTERM\r"); //stops http service
  delay(500);
  //Serial.print(mySerial.readString());
  mySerialRead();
  Serial.println("END TERM");
}

void setup()  {
  pinMode(rxp, INPUT);
  pinMode(txp, OUTPUT);
  Serial.begin(9600);
    while(!Serial){} 
  mySerial.begin(9600);
  delay(100);

  Serial.print("Starting the test");
}

void loop()
{   
  Serial.println("\n starting over\n");
  apiTest();  
  delay(3000);
}

Here is the response - I let it run for a few loops to show hopefully show a bigger picture:

Starting the test
 starting over

init

OK
END INIT
http string

OK
END HTTP STRING
http action

OK
END HTTP ACTION
read http data

ERROR
END READ
term

ERROR
END TERM

 starting over

init

ERROR
END INIT
http string

OK
END HTTP STRING
http action

+HTTPACTION: 0,200,351

OK
END HTTP ACTION
read http data

+HTTPACTION: 0,200,351

OK

+HTTPREAD: DATA,350
END READ
term
END TERM

 starting over

init
{"abbrev:null,"dst_offset":0,"dst_until":null,"raw_offset":-21600,"timezone":"America/Chicago","unixtime"
OK
END INIT
http string

OK
END HTTP STRING
http action

OK
END HTTP ACTION
read http data

ERROR
END READ
term

ERROR
END TERM

 starting over

init

ERROR
END INIT
http string

OK
END HTTP STRING
http action

+HTTPACTION: 0,200,351

OK
END HTTP ACTION
read http data

+HTTPACTION: 0,200,351

OK

+HTTPREAD: DATA,350
END READ
term
END TERM

 starting over

init
{"abbrevull,"dst_offset":0,"dst_until":null,"raw_offset":-21600,"timezone":"America/Chicago","unixtime":1
OK
END INIT
http string

OK
END HTTP STRING
http action

OK
END HTTP ACTION
read http data

+HTTPACTION: 0,200,351

OK

+HTTPREAD: DATA,350
END READ
term
END TERM

My concerns are with the response starting with "{"abbrevull"...
I would expect this to be sent immediately after the HTTP READ and prior to the HTTP TERM but somehow it doesn't get pushed through until after my loop restarts???
ALSO, here is what that API response is SUPPOSED to look like for reference...

{"abbreviation":"CST","client_ip":"2600:1700:6e0:c890:55c0:8c99:730c:b632","datetime":"2022-12-16T00:37:14.882338-06:00","day_of_week":5,"day_of_year":350,"dst":false,"dst_from":null,"dst_offset":0,"dst_until":null,"raw_offset":-21600,"timezone":"America/Chicago","unixtime":1671172634,"utc_datetime":"2022-12-16T06:37:14.882338+00:00","utc_offset":"-06:00","week_number":50}

Any help on this would be GREATLY appreciated!!! Thank you in advance!!

Sailing a bit close to the wind with 376 chars in the message, plus the CR and LF chars and any other strays.

Your idea seems ok for collecting the buffer, but I haven't loaded or run the code.

Maybe something like this...


char rxbuffer [400];		// global receive buffer
char rxPtr = rxbuffer		// initialise to point at rxBuffer

//----------------------------------------------------
// ONE CHARACTER into rxBuffer per pass
//----------------------------------------------------
void getSerial() {  // called if Serial.available() > 0
	char c;  // local character received
	//---------------------------
	c = Serial.read(); // get the char, and decrement the serial receive buffer
	//----------------
	switch (c) {
		// handle all the SINGLE character commands interactively
		// and capture longer commands to rxBuffer for processing at the end of the line (CR)

		// do whatever is needed with single keystrokes...
		case '?':
			break;
		//----------------
		case '@': // display the current motor status
			break;
		//----------------
		case '\r': // (CR) / ENTER / RETURN
			*rxPtr = 0; // add trailing terminator 
			parse(rxBuffer);
			rxPtr = rxBuffer; // clear the buffer
			break;
		//----------------
		case '\n': // NEWLINE
			// ignore it, do nothing
			break;
		//----------------
		case 27: // ESCAPE
			if (rxBuffer[0] > 0)
				Serial.println(F(" - CLEAR"));
			rxPtr = rxBuffer; // clear the buffer
			break;
		//---------------------------
		default:
			// save all other chars into the serial rxBuffer		
			if  (((c >= ' ') && (c < '~')) || (c == '\n')) {  
				Serial.print(c);	// local echo
				*rxPtr++ = c;
			}
	}
}
//----------------------------------------------------
void parse(char* commandstring) {
	// use strcmp, strtok and other c-string functionas to find and process your command verbs.
	
}

Thank you for the reply! I will give your code a shot and let you know the result. Let me know if you get a chance to run mine as well. Regarding the 376 chars, is there an obvious upper limit I should be aware of??

Only to keep in mind your available RAM, and not to block the cpu with nonsense code !

I used your commands and manually sent them to a SIM7600... via this sketch.

#include <SoftwareSerial.h>

#define BAUD 38400

SoftwareSerial softSerial (A1, A2);  // Rx, Tx as required.


void setup()
{
  Serial.begin(BAUD);
  softSerial.begin(BAUD);

  Serial.print("Ready @ ");
  Serial.print(BAUD);
  Serial.println(" baud");
}


void loop()
{ 
  while (Serial.available() > 0)
    softSerial.write(Serial.read());

  while (softSerial.available() > 0)
    Serial.write(softSerial.read());
}

I got the expected response as below...

21:02:23.979 -> AT+HTTPINIT
21:02:24.013 -> OK
21:03:37.814 -> AT+HTTPPARA="URL","http://worldtimeapi.org/api/timezone/America/Chicago"
21:03:37.814 -> OK
21:03:58.862 -> AT+HTTPACTION=0
21:03:58.862 -> OK
21:03:59.329 -> 
21:03:59.329 -> +HTTPACTION: 0,200,350
21:04:23.615 -> AT+HTTPREAD=0,350
21:04:23.615 -> OK
21:04:23.615 -> 
21:04:23.615 -> +HTTPREAD: DATA,350
21:04:23.615 -> {"abbreviation":"CST","client_ip":"49.224.84.168","datetime":"2022-12-16T02:03:59.387377-06:00","day_of_week":5,"day_of_year":350,"dst":false,"dst_from":null,"dst_offset":0,"dst_until":null,"raw_offset":-21600,"timezone":"America/Chicago","unixtime":1671177839,"utc_datetime":"2022-12-16T08:03:59.387377+00:00","utc_offset":"-06:00","week_number":50}
21:04:23.725 -> +HTTPREAD: 0
21:05:14.095 -> AT+HTTPTERM
21:05:14.095 -> OK

So the problem is in the way you are receiving the responses... and in particular some of your delays are slowing things down and you are therefore missing data.

Have a look at this version below... which is an adapted version of the basic sketch above that sends your commands at 2 seconds intervals... but importantly it is always listening for responses. You will need to adjust the Rx/Tx pins, and baud rate to match your module.

#include <SoftwareSerial.h>

#define BAUD 38400

SoftwareSerial softSerial (A1, A2);  // Rx, Tx as required.


void setup()
{
  Serial.begin(BAUD);
  softSerial.begin(BAUD);

  Serial.print("Ready @ ");
  Serial.print(BAUD);
  Serial.println(" baud");
}


unsigned long currentMillis;
unsigned long start = 0;
uint8_t i = 0;
String command[] = {"AT+HTTPINIT",
                    "AT+HTTPPARA=\"URL\",\"http://worldtimeapi.org/api/timezone/America/Chicago\"", 
                    "AT+HTTPACTION=0",
                    "AT+HTTPREAD=0,350",
                    "AT+HTTPTERM"};

void loop()
{
  currentMillis = millis();

  if (currentMillis - start > 2000)     // Send a command every 2 seconds
  {
    start = currentMillis;
    softSerial.println(command[i]);
    i = (i + 1) % 5;
  }
  
  while (Serial.available() > 0)
    softSerial.write(Serial.read());

  while (softSerial.available() > 0)
    Serial.write(softSerial.read());
}

... and here is the output.

22:44:35.912 -> AT+HTTPINIT
22:44:35.912 -> OK
22:44:37.909 -> AT+HTTPPARA="URL","http://worldtimeapi.org/api/timezone/America/Chicago"
22:44:37.947 -> OK
22:44:39.898 -> AT+HTTPACTION=0
22:44:39.898 -> OK
22:44:40.139 -> 
22:44:40.139 -> +HTTPACTION: 0,200,350
22:44:41.894 -> AT+HTTPREAD=0,350
22:44:41.894 -> OK
22:44:41.930 -> 
22:44:41.930 -> +HTTPREAD: DATA,350
22:44:41.930 -> {"abbreviation":"CST","client_ip":"49.224.71.235","datetime":"2022-12-16T03:44:40.024840-06:00","day_of_week":5,"day_of_year":350,"dst":false,"dst_from":null,"dst_offset":0,"dst_until":null,"raw_offset":-21600,"timezone":"America/Chicago","unixtime":1671183880,"utc_datetime":"2022-12-16T09:44:40.024840+00:00","utc_offset":"-06:00","week_number":50}
22:44:41.997 -> +HTTPREAD: 0
22:44:43.905 -> AT+HTTPTERM
22:44:43.905 -> OK

Hello red_car! Thank you for the reply! Yes, I was also able to get solid replies when attempting these commands one by one using PuTTy. Here is the response I get after running your code:

Starting the test
ERROR

OK

OK

+HTTPACTION: 0,200,351

OK

+HTTPREAD: DATA,350
{"abbreviation":"CST","client_ip":
OK

I had to add the "\r" at the end of each line of the command String but it appears to be responding properly! The next thing is for some reason it is still cutting the HTTPREAD result short...Any thoughts there? Thanks again!

You shouldn't need to do that if you are using println to send the commands.

Are you running the exact code I posted? ...with the exemption of the Rx/Tx and baud rates changed.

I note you don't get the command echoed back so your module must be set with ATE0... that's ok though.

Almost identical... But I removed the

  while (Serial.available() > 0)
    softSerial.write(Serial.read());

portion at first. Now that I put it back it is the same result...HOWEVER, I sometimes get this response: +HTTP_PEER_CLOSED
Whenever I don't get that error, the response is the same short version I mentioned above:

+HTTPACTION: 0,200,351

OK

+HTTPREAD: DATA,350
{"abbreviation":"CST","client_ip":

Let me know your thoughts on this? Thanks in advance!

Can't really explain it... I don't get this problem. I changed to 9600 baud and it still ran fine. I do get the occasional error responses... I guess this could be something server side, or network related.

Hello again!
I'm making progress! I kind of had to Frankenstein your suggestion along with some other stuff I found online to get to this point. Note that I did use your same String {command} idea but my code below, though it looks slightly different, returned the same results.
I am able to get larger chunks of the response now but stuff is getting sliced in weird places...I'm not sure why my responses aren't as "clean cut" as they should be. Any advice on how to more cleanly end the different responses would be greatly appreciated!

#include <Arduino.h>
#include "SoftwareSerial.h"
#include <stdio.h>
#include <string.h>

#define rxp 2
#define txp 3

SoftwareSerial mySerial = SoftwareSerial (rxp,txp); 
char replyBuffer[500];

unsigned long currentMillis;
unsigned long start = 0;
uint8_t i = 0;

String test1 = "AT+HTTPINIT\r";
String test2 = "AT+HTTPPARA=\"URL\",\"http://worldtimeapi.org/api/timezone/America/Chicago\"\r";
String test3 = "AT+HTTPACTION=0\r";
String test4 = "AT+HTTPREAD=0,350\r";
String test5 = "AT+HTTPTERM\r";

void newTest(String testCommand)
{
  mySerial.print(testCommand);
  uint8_t idx = 0;
  unsigned long timer = millis();

  while (millis() - timer < 3000) {
  	if (mySerial.available()) {
  	  replyBuffer[idx] = mySerial.read();
  	  if (replyBuffer[idx] != '\r')
      {
        idx++;
      }
    }
  }
  Serial.print(F("\t<--- ")); 
  Serial.println(replyBuffer);
}

void setup()  {
  pinMode(rxp, INPUT);
  pinMode(txp, OUTPUT);
  Serial.begin(9600);
    while(!Serial){} 
  mySerial.begin(9600);
  delay(100);
  Serial.print("Starting the test");
}

void loop()
{   
  newTest(test1);
  delay(3000);
  newTest(test2);
  delay(3000);
  newTest(test3);
  delay(3000);
  newTest(test4);
  delay(3000);
  newTest(test5); 
  Serial.println("\n starting over\n");
  delay(3000);
}

Here is the response:

Starting the test       <--- AT+HTTPINIT
ERROR

        <--- AT+HTTPPARA="URL","http://worldtimeapi.org/api/timezone/America/Chicago"
OK

        <--- AT+HTTPACTION=0
OK

+HTTPACTION: 0,200,351
api/timezone/America/Chicago"
OK

        <--- mezone":"America/Chicago","unixtime":1671248145,"utc_datetime":"2022-12-17T03:35:45.038206+00:00","utc_offset":"-06:00","week_number":50
+HTTPREAD: 0
k":5,"day_of_year":350,"dst":false,"dst_from":null,"dst_offset":0,"dst_until":null,"raw_offset":-21600,"ti
        <--- AT+HTTPTERM
OK
a/Chicago","unixtime":1671248145,"utc_datetime":"2022-12-17T03:35:45.038206+00:00","utc_offset":"-06:00","week_number":50
+HTTPREAD: 0
k":5,"day_of_year":350,"dst":false,"dst_from":null,"dst_offset":0,"dst_until":null,"raw_offset":-21600,"ti

 starting over

As you can see, I am getting longer bits of data now but they are just jumbled up. For example, my HTTPACTION response somehow has a piece of the URL?? Then the HTTPREAD response is cut in half almost and even seems to carry over into the HTTPTERM portion...I feel like this has to be something with me not "terminating" the responses properly?? Thanks again for continuing to help. I greatly appreciate it.

Ok... I think I finally figured this out !

8 bit unsigned... equals maximum 255 ! ... not so good when your buffer is 500 long.

Other minor things...

I wouldn't bother with this... some responses contain carriage returns... just include them... do this instead...

  	  replyBuffer[idx++] = mySerial.read();

... and you should also terminate the string before printing with...

  replyBuffer[idx] = '\0';

Also... you don't need these includes...

Boom!! That just about did it! It was the combination of changing the idx from uint8_t to an int, along with the terminating replyBuffer[idx] = '\0'; portion.

Thank you so much for your help!!

The only thing that is happening now is SOMETIMES my responses come in like this:

        <--- AT+HTTPINIT
OK

        <--- AT+HTTPPARA="URL","http://worldtimeapi.org/api/timezone/America/Chicago"
OK

        <--- AT+HTTPACTION=0
OK

        <--- AT+HTTPREAD=0,350
ERROR

        <--- AT+HTTPTERM
ERROR

        <--- 
+HTTPACTION: 0,200,351
AT+HTTPINIT
ERROR

Is this due to moving onto the next serial output too quickly?

EDIT: this issue doesn't happen every time as the last, say, 20 loops are all perfect...

Hard to say... I don't think so (because you are getting a response). I guess there might be a number of reasons the web server is not able to respond and you get an error... you can make your code more robust by handling these ERROR responses when you occasionally receive them. Good luck with your project..

Yeah I will definitely be adding in response checks to account for these types of situations. Thank you again!

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.