Issue with myGsm.readString( );

Projects Programming
1

Hello Arduino Community,

I have facing the issue in reading a string from GSM module.

Application is to fetch the data using http GET request from the server. Also successfully data reading for the first time fetch.

Issue is while reading the data for the second time, buffer is not reading the whole data (Please find the attached image)

Also code is attached for your reference.

static void printhttpdatastring( )
{
/* Here reading the whole data for the first time. From second fetch, data is half of the first fetch */
String httpDataString = myGsm.readString( ); // read the incoming data as string
delay( 1000 );
httpDataString.trim( );

Serial.print( httpDataString );
...
}

Look back to hear from you.

Thanks.

Arduino_error.png
Arduino_error.png1363×634 18.8 KB

weather_display_v4.0.ino (12.1 KB)

2

Serial is slow; the sender might still be in the process of sending and readString() can timeout if there is an short interruption and you will only have half a message. You can set a timeout using Serial.setTimeout()

You're possibly better off reading one character at a time and building up the String till you receive something like OK; I'm not that familiar with the lower level HTTP stuff to be sure what to wait for.

And you will be even better off not even using String (capital S) but nul-terminated character arrays.

3

sterretje:
Serial is slow; the sender might still be in the process of sending and readString() can timeout if there is an short interruption and you will only have half a message. You can set a timeout using Serial.setTimeout()

You're possibly better off reading one character at a time and building up the String till you receive something like OK; I'm not that familiar with the lower level HTTP stuff to be sure what to wait for.

And you will be even better off not even using String (capital S) but nul-terminated character arrays.

Thanks for your immediate response sterretje.

Did some changes and tried to take a byte at a time and store it in a char array. Still am reading the half of the response. Same baud rate for communication. Please follow the snippet, correct me if am wrong.

char serverResponse[280];
int index = 0;
char serverByte;

while(myGsm.available( )!=0)
{
serverByte = (char)myGsm.read();
serverResponse[index] = serverByte;
index++;
}
serverResponse[index] = '\0'; // Null terminate the string
Serial.println( serverResponse );

4

whats the end marker? Newline or carriage return?

Read into buffer until end marker.

5

Hello,

Thanks for your responses.

Figured out with below code. Working fine and reading all the information.

void read_UART_buffer(char *buffer,int count, unsigned int timeOut) {
  int i = 0;
  unsigned long timerStart;
  timerStart = millis();

 while(1) {
  while(myGsm.available()!=0){  
  
    char c = myGsm.read();
    HTTP_Respose_buffer[i++] = c;

  }
 
    if((unsigned long) (millis() - timerStart)  > timeOut * 1000UL) {
            break;
        }
  
 }
 HTTP_Respose_buffer[i]='\0';
 Serial.println( HTTP_Respose_buffer );

}

Below is the response am reading and storing in HTTP_Respose_buffer--

HTTPREAD: 221
[{"temp_Max":25.5,"temp_Min":25.3,"temp_Avg":25.3,"humidity":89.3,"wind_Direction":"249.000","wind_Speed":1.1,"rainfall":0.0,"wind_Max":0.0,"date":"2018-09-28","time":"17:40","stationname":"Gonzalves Training"}]
OK

Now how should i split the char array and store the required values in a buffer ?

Ex: String temp_Avg = "25.3", String humidity = 89.3

How to convert char array to string ?

6

Why do you want to go back to String?

Your output looks like jason if I'm not mistaken; if so, there is a jason library that might make life easy. I'm not familiar with jason.

Else you can use strtok() to split the character array on the comma and next check the individual parts. This should get you going, I haven't completely worked it out.

char HTTP_response_buffer[] = "[{\"temp_Max\":25.5,\"temp_Min\":25.3,\"temp_Avg\":25.3,\"humidity\":89.3,\"wind_Direction\":\"249.000\",\"wind_Speed\":1.1,\"rainfall\":0.0,\"wind_Max\":0.0,\"date\":\"2018-09-28\",\"time\":\"17:40\",\"stationname\":\"Gonzalves Training\"}]\nOK";

void setup()
{
  Serial.begin(57600);
  Serial.println("original");
  Serial.println(HTTP_response_buffer);
  // remove some stuff at the end
  char *p = strstr(HTTP_response_buffer, "}]");
  if (p == NULL)
  {
    Serial.println("Message does not end with }]");
    for (;;);
  }
  // remove the end from the message
  *p = '\0';
  // set p to the second position in HTTP_response buffer (skip [{)
  p = &HTTP_response_buffer[2];
  Serial.println("stripped");
  Serial.println(p);

  Serial.println();

  char *field = strtok(p, ",");

  while (field != NULL)
  {
    // print the fields
    Serial.println(field);
    field = strtok(NULL, ",");
  }
}

void loop()
{
  // put your main code here, to run repeatedly:

}

Output:

original
[{"temp_Max":25.5,"temp_Min":25.3,"temp_Avg":25.3,"humidity":89.3,"wind_Direction":"249.000","wind_Speed":1.1,"rainfall":0.0,"wind_Max":0.0,"date":"2018-09-28","time":"17:40","stationname":"Gonzalves Training"}]
OK
stripped
"temp_Max":25.5,"temp_Min":25.3,"temp_Avg":25.3,"humidity":89.3,"wind_Direction":"249.000","wind_Speed":1.1,"rainfall":0.0,"wind_Max":0.0,"date":"2018-09-28","time":"17:40","stationname":"Gonzalves Training"

"temp_Max":25.5
"temp_Min":25.3
"temp_Avg":25.3
"humidity":89.3
"wind_Direction":"249.000"
"wind_Speed":1.1
"rainfall":0.0
"wind_Max":0.0
"date":"2018-09-28"
"time":"17:40"
"stationname":"Gonzalves Training"

In a similar way as demonstrated, you can split the fields in key/value by splitting on the colon. Use strcmp() to check if a key is e.g. "wind_Direction" if you're only interested in a few fields or store it in specific variables.

You also might want to check if the first two characters are [{.

7

Hello,

Am trying with Json library. Hard coded Http response string with Json library is working fine. But when data read from serial and stored as buffer is no ouput. Below is the code

void JSON_parse()
{

  Serial.println("original");
  Serial.println(HttpResponsebuffer); //  Read from serialport
  // remove some stuff at the end
  char *p = strstr(HttpResponsebuffer , "]"); //removed the square brackets to form in proper Json
  if (p == NULL)
  {
    Serial.println("Message does not end with ]");
    for (;;);
  }
  // remove the end from the message
  *p = '\0';
  // set p to the second position in HTTP_response buffer (skip up to[)
  p = &buffer[19];
  Serial.println("stripped");
  Serial.println(p);

  Serial.println();


  // Memory pool for JSON object tree.
  //
  // Inside the brackets, 200 is the size of the pool in bytes.
  // Don't forget to change this value to match your JSON document.
  // Use arduinojson.org/assistant to compute the capacity.
  StaticJsonBuffer<250> jsonBuffer;



  // Root of the object tree.
  //
  // It's a reference to the JsonObject, the actual bytes are inside the
  // JsonBuffer with all the other nodes of the object tree.
  // Memory is freed when jsonBuffer goes out of scope.
  JsonObject& root = jsonBuffer.parseObject(p);

  // Test if parsing succeeds.
  if (!root.success()) {
    Serial.println("parseObject() failed");
    return;
  }

  // Fetch values.
  //
  // Most of the time, you can rely on the implicit casts.
  // In other case, you can do root["time"].as<long>();
 
  String time = root["time"];
  String Temperature = root["temp_Avg"];
  String Humidity = root["humidity"];
  String windDirection = root["wind_Direction"];
  String windspeed = root["wind_Speed"];
  String rainfall = root["rainfall"];

  // Print values.
 
  Serial.println(time);
  Serial.println(Temperature);
  Serial.println(Humidity);
  Serial.println(windDirection);
  Serial.println(windspeed);
  Serial.println(rainfall);
  
}
8 
  // set p to the second position in HTTP_response buffer (skip up to[)
  p = &buffer[19];

Wrong. That is NOT what that code does.

But when data read from serial and stored as buffer is no ouput.

And no proof. Not a shred.

Below is the code

Bullshit. That is PART of the code.

PART of the answer is:
You need to

To see ALL of the answer, post ALL of the code AND the serial output when you run the code!

9

Thanks for the response.

Figured out the issue. Now able to split the buffer and store in required variables with below code

char *Temperature;
char *Humidity;
char *WindSpeed;
char *WindDirection;
char *RainFall;

void SplitBuffer()
{
  Serial.println("original");
  Serial.println(buffer);
  // remove some stuff at the end
  char *p = strstr(buffer , "}]");
  if (p == NULL)
  {
    Serial.println("Message does not end with }]");
    for (;;);
  }
  // remove the end from the message
  *p = '\0';
  // set p to the second position in HTTP_response buffer (skip up to[{)
  p = &buffer[20];
  Serial.println("stripped");
  Serial.println(p);

  Serial.println();
  int i =0;
  char *field = strtok(p, ":");
  char *data[11];
  while (field != NULL)
  {
    // print the fields
    data[i++] = field;
  //  Serial.println(data[i]);
    field = strtok(NULL, ":");
  }

  Temperature = strtok(data[3],",");
  strcat( Temperature," degC");
  Serial.print("Temperature:");
  Serial.println(Temperature);

  delay(50);
  Humidity = strtok(data[4],",");
  strcat( Humidity," %");
  Serial.print("Humidity:");
  Serial.println(Humidity);

  delay(50);
  WindDirection = strtok(data[5],",");
  strcat( WindDirection," deg");
  Serial.print("WindDirection:");
  Serial.println(WindDirection);

  delay(50);
  WindSpeed = strtok(data[6],",");
  strcat( WindSpeed," Kmph");
  Serial.print("WindSpeed:");
  Serial.println(WindSpeed);

  delay(50);
  RainFall = strtok(data[7],",");
  strcat( RainFall," mm");
  Serial.print("RainFall:");
  Serial.println(RainFall);

 
}

Thanks for the support.
Best luck.

10

It would be orders of magnitude simpler to use sscanf() than all that strcpy() and strtok() crap.

11

Never thought of sscanf! In some cases like this one it really seems better to use it.

But its easy to right out of array boundaries if you are unsure that chars between delimiters will fit into array pointed to be written by sscanf.

12

But its easy to right out of array boundaries if you are unsure that chars between delimiters will fit into array pointed to be written by sscanf.

sscanf() can convert the strings to ints and (maybe) floats, so there is no reason to deal with strings for temperature, humidity, wind speed, etc.

13

Thats if incoming data is sent obeying parsing rules.
If data sent from user is being parsed that way and if wind speed is Sent as alphanumeric i.e not number then there is no need to continue parsing leftover data to variable addresses as sscanf does.

Via strcmp,strstr and strtok is possible to break the action right away without parsing all the “wrong” data.

Am not sure though if sscanf will continue or will stop parsing never tested.

example:

#include <stdio.h>
#include <errno.h>
#include <string.h>


    int main(int argc, char *argv[]) {
	
        char a[]="123:555:444";
        int b;
        int c;
        int d;
        
        sscanf(a,"%d%*c%d%*c%d",&b,&c,&d);
        
        printf("%d,%d,%d",b,c,d);
        
	return 0;
}

output:

123,555,444
Time elapsed: 000:00:031
Press any key to continue

Now assuming someone played with input string and put alphanum in url string:
code is same except: char a[]="123:12anc:444"; 555 changed to 12anc

output:

123,12,0
Time elapsed: 000:00:015
Press any key to continue

which is preventable wth strcmp/strstr and strtok.