Another solution than increasing the buffer size for arduino

Using Arduino Programming Questions
1

Hi all,

I'm doing a project where i have to send an HTTP request to my GSM module through arduino. When I use the serial monitor to manually send the commands everything goes smoothly but whenever I try to let the arduino send these commands by its own, one command don't get send and I think the reason is the large size of the command

here is the code I wanna send
(I included only the parts for the communication with the module since my code is quite long


SoftwareSerial MYGSM(7, 8); //Define virtual serial port name as MYGSM,Rx is port 7, Tx is port 8


void sendGSM(const char* msg) { //a function that communicated with the GSM 
  MYGSM.println(msg);
}

//somewhere inside my loop().......
    char *first_part = "AT+HTTPPARA=\"URL\",\"https://smartbreathalyzer.000webhostapp.com/GETID.php?TakenID=";
    char *closing = "\"";
    String url = first_part + IC + closing; //the IC here is a number that the user will input through a keypad 
    const char *url_complete = url.c_str();

// the commands to be sent 

      sendGSM("AT+HTTPTERM"); //to make sure it is off first
      delay(100);
      sendGSM("AT+HTTPINIT");  //START HTTP
      delay(100);
  //    Serial.println(url_complete);
      sendGSM(url_complete);  //SEND THE LINK
      delay(100);

as you can see the url_complete is quite long

After googling for a while, I came out to a solution that is basically increasing the buffer size from 64 to 256 BUT that is going to take too much size in the SRAM and when I try to use the F() function to store the variables to the flash memory I get this error:

cannot convert 'const __FlashStringHelper*' to 'const char*' for argument '1' to 'void sendGSM(const char*)'

I add the F() function like this:

    sendGSM(F(url_complete));  //SEND THE LINK

Even when I try to place the F() function at other locations like:

const char *url_complete = F(url).c_str();

I still get errors

So, any suggestions to solve this issue?
Also, I think when the module communicates back to the Arduino it would also carry large statements so probably I need to somehow find a way to allow these large statements to happen (The buffer size increment could solve it, however I don't want to risk using lots of SRAM)

EDIT: I SAW ON GOOGLE, that Flash (PROGMEM) memory can only be populated at program burn time. You can’t change the values in the flash after the program has started running.
It seems bad idea to use it in my case since every time I will have a different IC, right?

2

Is there a specific reason for the sendGSM function, instead of using MYGSM.println? It would be easier to accomplish if you could use both print and println (alternatively you could use print in the sendGSM function, and append \r\n to the text to send the carriage return and linefeed).

There is no need to concatenate the entire text into a single string.

  // the commands to be sent

  MYGSM.println(F("AT+HTTPTERM")); //to make sure it is off first
  delay(100);
  MYGSM.println(F("AT+HTTPINIT"));  //START HTTP
  delay(100);
  //    Serial.println(url_complete);
  MYGSM.print(F("AT+HTTPPARA=\"URL\",\"https://smartbreathalyzer.000webhostapp.com/GETID.php?TakenID="));
  MYGSM.print(IC);
  MYGSM.println(F("\""));
  delay(100);

There really is no need to increase the buffer size, I would suspect your problem is that you are already using too much RAM by creating all those strings, then concatenating them into a separate String.

2 Likes
3

You're right the sendGSM function is just an extra thing.

The F() has worked but now I got another issue

But first, let me show you this:

//parse machine with different states
enum parseState {
  PS_DETECT_MSG_TYPE,//to detect the type of the msg
  PS_IGNORING_COMMON_ECHO,// here we ignore those we don't need
  PS_HTTPACTION_TYPE, // here we deal with responcde related to http action
  PS_HTTPACTION_RESULT, // the result of the httpaction, usually comes like 0,200,64, the 64 is the length
  PS_HTTPACTION_LENGTH, // here we tell the arduino the length of content is 64 from above example
  PS_HTTPREAD_LENGTH, // here we show the length of the content
  PS_HTTPREAD_CONTENT // here we print the content
};
byte parseState = PS_DETECT_MSG_TYPE; //start with this state
char buffer[256]; // buffer of character
byte pos = 0; // position of buffer
int contentLength = 0; // length of content
int SIZE = 0; //to save the size of data came from the HTTP

// now inside my loop() 

 if ( smallstate == correct ) { 
//my code is quite long and I'm handling many things, and this server communication starts here at the state

      lcd.setCursor(0, 0);
      lcd.print("Connecting to server.....");
      smallstate = parsing;
    }
    ////////////////////////////////
    else if ( smallstate == parsing ) { 
//thanks to you I used the F() here and it did save up space in my software 

      MYGSM.println(F("AT+HTTPTERM")); //to make sure it is off first
      delay(100);
      MYGSM.println(F("AT+HTTPINIT"));  //START HTTP
      delay(100);
      MYGSM.print(F("AT+HTTPPARA=\"URL\",\"https://smartbreathalyzer.000webhostapp.com/GETID.php?TakenID="));  //SEND THE LINK
      MYGSM.print(IC);
      MYGSM.println(F("\""));
      delay(100);
      MYGSM.println("AT+HTTPACTION=0"); //CHOOSE GET METHOD
      delay(100);
      smallstate = afterparsing;
    }
    else if (smallstate == afterparsing) {
      while (MYGSM.available()) {
        parseATText(MYGSM.read());  //this is a simple machine to show on my serial monitor what is happening 
      }
    }


void parseATText(byte b) {
  buffer[pos++] = b;
  if (pos >= sizeof(buffer))
    resetBuffer(); //just to be safe
  /*//Detailed debugging
    Serial.println();
    Serial.print("state = ");
    Serial.println(parseState);
    Serial.println("b = ");
    Serial.print("pos = ");
    Serial.println(pos);
    Serial.print("buffer = ");
    Serial.println(buffer);*/
  switch (parseState) {
    case PS_DETECT_MSG_TYPE:
      if (b == '\n')
        resetBuffer();
      else {
        if ( pos == 3 && strcmp(buffer, "AT+") == 0 ) {
          parseState = PS_IGNORING_COMMON_ECHO;
        }
        else if ( b == ':') {
          //Serial.print("Checking message type:   ");
          //Serial.println(buffer);

          if (strcmp(buffer, "+/HTTPACTION:") == 0) {
            Serial.println("Received HTTPACTION");
            parseState = PS_HTTPACTION_TYPE;
          }
          else if (strcmp(buffer, "+HTTPREAD:") == 0) {
            Serial.println("Received HTTPREAD");
            parseState = PS_HTTPREAD_LENGTH;
          }
          resetBuffer();
        }
      }
      break;
      
    case PS_IGNORING_COMMON_ECHO:
      {
        if (b == '\n') {
          Serial.print("Ignoring echo:  ");
          Serial.println(buffer);
          parseState = PS_DETECT_MSG_TYPE;
          resetBuffer();
        }
      }
      break;
      
    case PS_HTTPACTION_TYPE:
      {
        if ( b == ',') {
          Serial.print("HTTPACTION type is ");
          Serial.println(buffer);
          parseState = PS_HTTPACTION_RESULT;
          resetBuffer();
        }
      }
      break;

    case PS_HTTPACTION_RESULT:
      {
        if (b == ',') {
          Serial.print("HTTPACTION RESULT (200 IS GOOD) is ");
          Serial.println(buffer);
          parseState = PS_HTTPACTION_LENGTH;
          resetBuffer();
        }
      }
      break;
      
    case PS_HTTPACTION_LENGTH:
      {
        if (b == '\n') { //WHEN WE REACH THE END OF LINE FROM THE RESPONCE X.XXX.XX
          Serial.print("HTTPACTION length is ");
          Serial.println(buffer); //SUPPOSEDLY GET THE LAST XX
          //now request content
          MYGSM.println("AT+HTTPREAD=0,");
          MYGSM.print(buffer); //PUT THE XX HERE
          parseState = PS_DETECT_MSG_TYPE;
          resetBuffer();
        }
      }
      break;
      
    case PS_HTTPREAD_LENGTH:
      {
        if (b == '\n') {
          SIZE = atoi(buffer);
          Serial.print("HTTPREAD length is  ");
          Serial.println(SIZE);

          Serial.print("HTTPREAD content:   ");
          parseState = PS_HTTPREAD_CONTENT;
          resetBuffer();
        }
      }
      break;

    case PS_HTTPREAD_CONTENT:
      {
        //here i can show the content length for demo
        Serial.write(b);
        SIZE--;
        if (SIZE <= 0 ) {
          // all content are read already
          parseState = PS_DETECT_MSG_TYPE;
          resetBuffer();
        }
      }
      break;
  }
}

Now let me show you the serial monitor output:

SM
SM711×137 2.16 KB

as you can see, for some reason the link is not being executed or idk what's happening, if I use Serial.println() for the same stuff it will show up..
Any idea what's happening?

4

when i replace this

      MYGSM.print(F("AT+HTTPPARA=\"URL\",\"https://smartbreathalyzer.000webhostapp.com/GETID.php?TakenID="));  //SEND THE LINK
      MYGSM.print(IC);
      MYGSM.println(F("\""));

with this

 MYGSM.println(F("AT+HTTPPARA=\"URL\",\"https://smartbreathalyzer.000webhostapp.com/GETID.php?TakenID=012345678\""));  //SEND THE LINK

everything works find

which might mean the module doesn't accept incomplete statements like print, print, and then println

but now I'm adding the IC number manually, I still need the software to automatically determine IC from the input and then add it to the link

5

what exact type of microcontroller are you using?

What baudrate are you using for software-serial?
software-serial is know to work reliable only for low baudrates like 9600 or 19200

The code-snippets you have posted show a lot of delays
it might be that these delays cause the problems.

And really ? increasing a buffer from 64 bytes to 256 bytes is a SRAM-problem?
256 - 64 = 192 Byte. Not that much.

You want to process a relatively big data. If the data becomes too big a bigger SRAM might be needed.

best regards Stefan

1 Like
6

Arduino UNO and SIM7600CE shield by DFrobot,

Baudrate is 19200

Even with removing the delays and replacing them with millis() or even without any timing I still have the problem

Increasing the buffer has even showed me an error when uploading the code to the arduino saying there is not much memory which could result in instability

A bigger SRAM means an Arduino mega, but that's a lot of pins, the UNO is more than enough for me if I manage to solve this issue somehow

Thanks

7

There are quite a number of different microcontrollers that can be programmed with the arduino-IDE that are even smaller in physical size and cheaper than an arduino-Uno but have plenty of SRAM.

3.3V-device

Powerful CPU: ARM® Cortex®-M0+ 32bit 48MHz microcontroller(SAMD21G18) with 256KB Flash,32KB SRAM.

image
image1024×696 65.4 KB

best regards Stefan

1 Like
8

At 19200 baud, the module will never see any difference between those. How is IC declared, and what is its value?

Its hard to tell what your other problem might be, I'm not familiar with the GSM module and the expected responses, but if it is responding after every AT command the receive buffer may be overflowing because of all the delays.

With an UNO, I would be concerned with the memory usage, you only have a small amount of RAM and the code you posted is using a lot for printing to the serial monitor.

1 Like
9

IC is declared as a string. Here is the code i use to get the IC

String IC;

    if ( smallstate == toTakeID ) {
      char key = kpd.getKey();
      if (key) {
        Serial.println(key);
        lcd.print(key);
        if (key >= '0' && key <= '9') {     // only act on numeric keys
          IC += key;               // append new character to input string
        } else if (key == '#') {
          if (IC.length() > 0) {
            lcd.clear();
            smallstate = correct;
          }
        } else if (key == '*') {
          IC = "";                 // clear input
          lcd.clear();
          lcd.setCursor(0, 0);
          lcd.print("Please enter the id ");
          lcd.setCursor(0, 1);
          lcd.print("Press # to confirm ");
          lcd.setCursor(0, 2);
          lcd.print(" and * to clear");
          lcd.setCursor(0, 3);
        }
      }
    }

The value is going to be either 10 or 12 digits, but anyway I'm treating it as a text since I'm gonna send it to the HTTP only.

it does respond with OK after each command except the HTTPREAD=0,xx, this will give back the server response which is just a name

That's true but I'm using all these serial prints just while making the program, later when I implement it I won't need them at all and I will be parsing the last response into a variable to be displayed on a LCD

10

If you have also printed the statement to the console and it is built correctly, the problem must be elsewhere.
After sending a long line, such as that HTTP GET statement, are you waiting long enough for the GSM device to process it? 100ms seems short for that command since it must wait for a response from the remote server. How long did it take to respond when you entered the command manually via the serial console? 100ms would appear instantaneous.

1 Like
11

It's a good bit of it! C++ String is a RAM-waster that matters in small memory environments.

Not putting constant text into PROGMEM and printing it from there?

If you send the message 1 char at a time, it will buffer chars on your board since serial only gets sent 1 char at a time wayyyyyyyyy slower than the buffer fills. The send buffer on your board is where the message assembles, there is NO NEED to "build the whole string to send it" at all --- it's a waste of RAM and cycles.

1 Like
12

You're right!

I did modify the code, but instead of timing it using delay or millis(); I just included the commands in the parsing machine so that I execute the command only when I get the response from the previous one. The only thing to do is to do the commands manually and then capture the responses into a notepad or something and make the code accordingly.

Also, using the F() like this worked

I guess the whole thing was about making the commands excursion related to the completion of the previous command.

Anyway, thanks to all of you guys for commenting and discussing.

13

Another RAM saving thing is to replace every

int

used for values 0 to 255 (ex: pin numbers) with

byte (Arduino for uint_8)

and save 1 byte each.

1 Like
14

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.