Need larger "String buffer", and, How to delete 1st 65 chars of a string

EDIT - I posted full code and more details - see post #4. :)

  1. Reading serial data from a GPRS modem to display on an LED sign. It works, until I send a text > 64 characters. Since SMS text messages can be up to 140 characters, I need to learn how to increase the String buffer from (what looks like) 64 characters to about 210 characters. That will hold the header and largest valid txt message. In my UNO R3, the string with messages are truncated at 63 characters + the ending null. I've read much on this but, frankly, It's not making sense to me and WAY over my head.

  2. How does one delete a specific number of characters from the beginning of a string of ascii data? I'f tried "replace" and "trim", but they don't accomplish the task. I know that the first 65 characters of the txt message needs to be deleted because it is the header of the text messages. I want to capture the entire string coming in from the GPRS/GSM modem into StringOne so the timestamp and originating phone number can be exploited in the future. To print the txt msg I need a copy of StringOne, (call it StringTwo) and delete the header to leave only the interesting/useful data being the txt message the originating user sent.

Thanks in advance!

It would not normally be necessary to make the serial buffer big enough to hold your entire message, since the Arduino can read the incoming text far quicker than it can be transmitted over the serial port. If your message is getting truncated that suggests you are not reading the serial port frequently enough, which points to a design flaw in the sketch.

If you don’t want to save the first 65 character of the text then don’t save them.

Sorry the advice isn’t very detailed, but since you haven’t posted your code it’s hard to be more specific.

Thanks for responding. Good point about my not posting code. Actually a VERY good point! I’ll get to the other computer and post the code. Sorry about that.

Here’s the full code. The “replace” commands on lines 96 and 97 are to get rid of the “cr lf OK cr lf” response form the modem AT command to read the tzt message. To delete the header line of the text, the best solution I could figure out was to empty the string at the “-” (minus sign) because it is always in the same index location and near the end of the header line. Following is a copy of an entire SMS message as reported from the GPRS/GSM modem. BTW, this is a SEEED Studio version 1.4. GPRS shield. OH. Deleting everything up to the “-” symbol, which is index# 58, proved to be a failed strategy. I learned this the first time I included telephone number in a text message sent to test the thing out… It deleted everything up to the last “-” sign… ooops. Keep in mind I am new at this stuff. But am now addicted.

To read txt message #1, issue the command AT+CMGR=1\r

Trying to figure out what the hek is happening, I made the following data sheet. The top line is the index of data in the string. The middle line is the actual response from the modem, header and txt message. The bottom line is the hex values of the ascii characters. (I did not post these questions without thoroughly investigating the matter until my brain broke… (I changed the telephone number to protect the innocent)

Respnse to command: AT+CMGR=1
(sorry that the spacing does not line up here. Copy it to notepad and it will line up nicely.)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
cr lf + C M G R : SP " R E C SP U N R E A D " , " + 1 5 5 5 1 2 3 4 5 6 7
0D 0A 2B 43 4D 47 52 3A 20 22 52 45 43 20 55 4E 52 45 41 44 22 2C 22 2B 31 35 35 35 31 32 33 34 35 36 37

35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69
" , " " , " 1 3 / 0 5 / 1 9 , 1 3 : 4 7 : 1 8 - 2 0 " CR LF C a l l SP Y
22 2C 22 22 2C 22 31 33 2F 30 35 2F 31 39 2C 31 33 3A 35 36 3A 30 31 2D 32 30 22 0D 0A 43 61 6C 6C 20 79

70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90
o u sp b a c k sp l a t e r cr lf cr lf O K cr lf
6F 75 20 62 61 63 6B 20 6C 61 74 65 72 0D 0A 0D 0A 4F 4B 0D 0A

Just realized this text message is not too long for the buffer(s). It was used to digest the header detail. The TXT message in this example is:
Call You back later cr lf

MUCH of the code can likely be deleted or condensed but I’m afraid to touch it since it works. Except for the buffer and delete header issues.

#include <SoftwareSerial.h>

SoftwareSerial GPRS(7, 8);
SoftwareSerial BetaBrite(2, 3);
String theMsgStuff= "\r\nSomething Displayed\r\n";
//String newmsg= ("starup message");
String currentLine = "";
String theString = "";
boolean readingmsg = false;
unsigned char buffer[400]; // buffer array for data recieve over serial port
//int count=0;     // counter for buffer array 


void setup(){

  pinMode(2, INPUT);
  pinMode(3, OUTPUT);
  pinMode(4, INPUT);
  pinMode(5, OUTPUT);
  pinMode(7, INPUT);
  pinMode(8, OUTPUT);
  UCSR0C = (0<<UPM00)|(1<<USBS0)|(3<<UCSZ00)|(0<<UCPOL0);//configure for 8N1

  GPRS.begin(9600); // Start the SoftwareSerial wich is using the GPRS UART
  Serial.begin(9600); // Start the HardwareSerial wich is using the Arduino UART ( DEBUG OR WHEN CONNECTED TO A COMPUTER  .. )
  delay(1000);
  Serial.println();
  Serial.println("[ DEBUG ] Void Setup Complete");
}

void loop(){
  handleIncomingMessages();
}
void handleIncomingMessages(){
  // initMessageRead();
  String content = "";
  // Serial.println("[ DEBUG ] Begin Sketch");
  if( GPRS.available() ){ // if new data is coming from the GPRS UART
    while(GPRS.available())          // reading data into char array 
    {
      char inChar = GPRS.read();
      content += inChar;
    }
  }
  else {
  }
  if( content != "" ){
    delay(100);
  }
  int smsNum = content.indexOf('+');

  if (content.charAt(smsNum + 8) == 'S') {
  handleMessage( content );   
  }
}
void handleMessage(String content){
  int smsNum = content.indexOf('+');
  if (content.charAt(smsNum + 8) == 'S') {

    //      Serial.println("[ DEBUG ] The SIM900 received a new message");
    delay(500);
    //   String content = "";
    //   handleIncomingMessages();  /// **************** LOOPS BACK TO TOP ******
    initMessageRead();
  } 
  else {
    Serial.println( "Was Not A New SMS" );
    delay(100);
    initMessageRead();
  }
}
// ************************ BEGIN READ MESSAGE *******************************
void initMessageRead(){
  String newMsg = "";
  String str = "";
  GPRS.begin(9600);
  GPRS.println("AT+CMGR=1\r");
  delay(700);

  if( GPRS.available() ){ // if new data is coming from the GPRS UART
    while( GPRS.available() ){ // while there's still data in serial ..
      char wholeMsg = (char)GPRS.read(); 
      newMsg += wholeMsg; 
      if (wholeMsg == '-') {
        newMsg = "";

      } 
    }

  }
  //      Serial.println("[DEBUG} Delete 1st line");


  if( newMsg != "" ){
    Serial.println(newMsg.length());
    newMsg.replace("OK\r\n", "");
    newMsg.replace("20\"", "");
    newMsg.trim();
    delay(10);
    //Serial.println(newMsg.length());
    Serial.print( newMsg );
    delay(500);
    UCSR0C = (2<<UPM00)|(0<<USBS0)|(2<<UCSZ00)|(0<<UCPOL0);//configure for 7E1
    SoftwareSerial BetaBrite(2, 3); // RX, TX
    BetaBrite.begin(9600);
    for(int i = 0; i < 5; i++) BetaBrite.write((byte)0x00); // auto detect baud
    BetaBrite.write((byte)0x01); // SOH
    BetaBrite.print("Z00"); // All displays
    BetaBrite.write((byte)0x02); // STX
    BetaBrite.print("AA"); // Text string
    BetaBrite.write((byte)0x1B); // ESC
    BetaBrite.write((byte)0x20); // Line number
    BetaBrite.print("a"); // Scroll right to left
    BetaBrite.write((byte)0x10); // New Line
    BetaBrite.print(" ");
    BetaBrite.print(newMsg);
    BetaBrite.write((byte)0x04); // ETX
    UCSR0C = (0<<UPM00)|(1<<USBS0)|(3<<UCSZ00)|(0<<UCPOL0);//configure for 8N1
    delay(200);
//    deleteIncomingMessages();
    //    Serial.println("[ DEBUG ] Message content from initMessageRead");
    //    Serial.println("[ raw message start ] " + newMsg + " [ raw message end ] " );
  }
  //else 
  deleteIncomingMessages();
}


void deleteIncomingMessages(){
  GPRS.begin(9600);
  GPRS.print("AT+CMGD=1,4\r"); // write the "delete all messages" request to the GPRS UART
  //     GPRS.write((byte)0x10);
  //     GPRS.write((byte)0x13);
  delay(2000); // give it a little bit of time to execute ( and normally return '+CMGR:' )
//  Serial.println("END");
  delay(2000);
}

Peter - If I knew how to "not save the first 65 characters in the first place", half the battle would be won. If that were the case now, I'd let the task of examining the header for a day when I have more experience at this.

REading some examples from Zoomcat I realize it's not "serial buffers" I need to increase. It's "String Buffer that needs to increase. Will modify the title and original question.

Will look at your code in more detail later, but my initial thoughts:

Don't use the String class. It exposes a memory corruption in older IDE versions and can lead to memory fragmentation in all versions; it's simply not sensible to use dynamic memory allocation to hold dynamic data on a platform with such limited memory and such limited ability to manage that memory. I recommend that you use plain old c-strings (null-terminated char arrays) instead.

Your comments about being afraid to touch the code make me wonder whether you are following good working practices. Make sure you back up your working code whenever you reach a useful milestone, so that you can always go back to a previous version. If you don't already have any tools or working practices in place for that, I recommend you look at the File Hampster automatic versioning tool. Also, don't be afraid to create test sketches specifically to work out how to solve some small part of your problem, and throw them away (and use the solution in your real working code) when you're finished with them. By the time you get to the point of asking for help on the forum, you should expect to have produced a minimal sketch that demonstrates the problem and your best attempt to solve it. Nine times out of ten, the act of creating this will be enough for you to solve the problem on your own. The tenth time, you have saved everyone on the forum wasted effort trying to understand code that is not relevant to the problem.

Peter - Your advice, every bit of it is good advice. I've started copying the sketch to new names manually, but will download the Hamster Version'ing tool. Good idea. You are likely correct about the string class issues. I just sent a very long txt msg to debug and stranger things are happening. A group of characters in the center of the string is "missing". The numbers 31 appear in the string which are not contained in the original text msg. No idea where they came from. Hard to describe so it's copied below: Thanks for your advice and assistance. This is a person project so there is no rush. It's all about the "learning"!

Following is the original test txt message sent to the GPRS/GSM modem:

******************************** BEGIN ORIGINAL TXT MESSAGE **************************** This is a long text message to check the string buffer size because it appears the buffer is set at 64 characters by default. However a tex msg can contain 140 characters ******************************** END ORIGINAL TXT MESSAGE ****************************

Following is the output on the terminal monitor: for commands -- Serial.println(newMsg.length()); -and- Serial.print(newMsg);

******************************** BEGIN OUTPUT ON TERMINAL MONITOR **************************** [ DEBUG ] Void Setup Complete68

This is a long text message to check the string buffer size bec31

ain 140 characters ******************************** END OUTPUT ON TERMINAL MONITOR ****************************

To "see" the behavior better I used the following txt of numbers (ten each) so as to be able to count visually:

111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999000000000011111111112222222222223333333333

The HEX output on the terminal monitor for Serial.println(newMsg.length()); 36 38 0D 0A 0D 0A

The HEX output on the terminal monitor for the string message is: 31 31 31 31 31 31 31 31 31 31 32 32 32 32 32 32 32 32 32 32 33 33 33 33 33 33 33 33 33 33 34 34 34 34 34 34 34 34 34 34 35 35 35 35 35 35 35 35 35 35 36 36 36 36 36 36 36 36 36 36 37 37 37

Get rid of this:

  if( content != "" ){
    delay(100);
  }

At 9600 baud you can be getting ~960 characters per second - your 10th second delay allows you to miss ~96 - more than enough to overflow the buffer.

OK. We’re getting somewhere. The delay you referenced is in the section that waits for the GSM modem to issue the short notice that a new msg has arrived. That section works flawlessly.

However, further down the sketch, on line #93, there is a delay(1000); after reading the larger SMS message. I commented that out and loaded the sketch. That did not work. It appeared that the UNO was getting overloaded and rebooting itself. HOWEVER! When I uncommented the delay(1000); and loaded it back into the UNO, it started working properly. The entire messages are being displayed on the serial monitor and printed out to the BetaBrite sign.

Not much confidence here right now… Makes no sense why it did not work with sketch A. Then commented out a delay creating sketch B. And it broke completely. Then put the delay back in, returning to Sketch A, and it works correctly. I’ve sent three 130 character SMS TXT messages to the GSM modem since uncommenting (putting the delay back in) and each one has worked properly.

I think you’re on to something and that the root of the problem is in the delays/timing.

You should not need to use delay() to process serial input or to communicate with a modem. If your code is only works with the ‘right’ delay then it is working by coincidence and you need to correct that. You should be able to detect and process serial input on any of your input streams when it is available and not make any assumptions about when it will arrive. Given the symptoms you describe could be caused by memory corruption, and the String class is a known cause of that, I suggest you make getting rid of that your priority.

I agree. There is no confidence in Sketch as it is now. Even though I've sat here for the last several minutes sending large msgs one after the other, without fail, it makes no sense.

I found the delay was necessary so the sketch did not proceed until all data has arrived. Earlier versions of the sketch without a delay would see one character come in, the sketch process it, and repeat. The terminal monitor had a lf cr and grew in length one character at a time until the entire message arrived. something like:

T Th Thi This This This i This is This is t This is th This is the etc etc etc.

I'm going to Barnes and Noble and find a book on Arduino sketches. Have been googling the c-array instead of the string method, but nothing on my level shows up. I've reached my level of incompetence. Which is very humbling being that I've jsut gotten started with programming and Arduino. p.s. I'm an old network systems engineer type guy. Been at it since the 70's when we had separate keyboard keys for cr and lf. And the output was paper and ticker tape.

Sorry, I find the statement "I found the delay was necessary so the sketch did not proceed until all data has arrived" illogical. (And I am being polite here, I have a stronger opinion than that ]:) )

Lets look at this fragment of code (and I have not examined all of it, I am just trying to pinpoint your misunderstanding about the delay())

 GPRS.println("AT+CMGR=1\r");
  delay(700);

  if( GPRS.available() ){ // if new data is 
    while( GPRS.available() ){ // while there's still data
      char wholeMsg = (char)GPRS.read(); 
      newMsg += wholeMsg; 
      if (wholeMsg == '-') {
        newMsg = "";
      } 
    }

You are assuming "all" the message has arrived in the serial buffer (which is, by the way, 64 bytes long... :astonished:) after exactly 700ms. Your while loop empties the (64 byte) buffer into the newMsg string. And then your code goes totally away, never to add more characters to the newMsg. If you have only a short delay - say 20ms - you have discovered that your code then copies the 20 odd characters that have arrived in that time, but you never go back looking if more arrive a little later. So you incorrectly concluded the delay is necessary.

You need to get rid of the delay, whether 700 or 20ms long. If there is no delay the code will immediatly check for the GPRS.available(). There may no be any character avialable yet or maybe only the 1st character. Your logic indicates that you use the "-" character to determine EndOfMessage (EOM). So your while test should be: as long as you have not seen that EOM character then keep looking for GPRS.available(). If that is false, keep waiting until it is true, add the next character to newMsg and if it is the "-" then stop, as you have the complete message, else go back to the GPRS.available() check. This is a long winded explanation, the code will "Boil down to" one while statement with two logic tests and the newMsg accumulation in the body of the while.

And as others have said: get rid of the String objects, but you can do that as a seperate exercise, later.

Sample code to read and buffer input from a serial port using c-strings has been posted on the forum numerous times and is not complex. Here’s one example which should give you the general idea:

// incomplete, untested
void handleSerial()
{
    const int BUFF_SIZE = 32; // make it big enough to hold your longest command
    static char buffer[BUFF_SIZE+1]; // +1 allows space for the null terminator
    static int length = 0; // number of characters currently in the buffer

    if(Serial.available())
    {
        char c = Serial.read();
        if((c == '\r') || (c == '\n'))
        {
            // end-of-line received
            if(length > 0)
            {
                handleReceivedMessage(buffer);
            }
            length = 0;
        }
        else
        {
            if(length < BUFF_SIZE)
            {
                buffer[length++] = c; // append the received character to the array
                buffer[length] = 0; // append the null terminator
            }
            else
            {
                // buffer full - discard the received character
            }
        }
    }
}

void handleReceivedMessage(char *msg)
{
	if(strcmp(msg, "on") == 0)
	{
		// handle the command "on"
	}
	else
	{
		// etc
	}
}

There may be some flawed logic in the discussions. One solution given requires a carriage return/line feed to work. Does your GPRS provide these as delimiters in its strings? If there are no end of packet markers sent by the GPES, then the given solution is N/A. Below is some simple code that uses a short delay to prevent emptying the serial input buffer before more bytes arrive. A simple example showing that the arduino can empty the serial input buffer much faster than data can arrive. Send your string from the serial monitor with the delay commented out, then do the same with the delay. Post an accurate example of the string you are receiving from the GPRS so that it can be checked for possible solutions.

// zoomkat 7-30-11 serial I/O string test
// type a string in serial monitor. then send or enter
// for IDE 0019 and later

String readString;

void setup() {
  Serial.begin(9600);
  Serial.println("serial test 0021"); // so I can keep track of what is loaded
}

void loop() {

  while (Serial.available()) {
    //delay(2);  //delay to allow byte to arrive in input buffer
    char c = Serial.read();
    readString += c;
  }

  if (readString.length() >0) {
    Serial.println(readString);

    readString="";
  } 
}

The GPRS does send \r\n at the beginning and end of the ascii data sent to the Arduino. If I comment out the delay it seems as if the Arduino locks up and then reboots. I believe this is happening because there is a test print string in the SETUP just before void loop, and it prints out again. And again. But when I put the delay statement back in (by removing the two comment slashes) and reloaded the UNO, it started working properly. The entire long string show up on the terminal monitor, and the string.length is well above 100, and the BetaBrite displays the entire string. I've sent dozens of 120+ character txt messages to it this afternoon and every one has displayed properly. There were no other changes in the sketch other than to comment out the delay, reload the sketch. And then uncomment the delay and reload the sketch. This is why I have a low level of confidence that this current config will be reliable.

I'll get some terminal monitor printouts and post here. But the behavior of the system has unexpectedly changed.

zoomkat: If there are no end of packet markers sent by the GPES, then the given solution is N/A.

If this is a reference to the code snippet I posted then I disagree. The code demonstrates how to receive a sequence of characters from a serial port and buffer them in a c-string, and this applies regardless of whether the end of each message is indicated by a newline or by some other condition. The main points are:

  • Read a character when it is available.
  • Buffer it in a null-terminated char array.
  • Don't use delays to control the rate of reading characters or to try to arrange for the sketch to only check the serial port when a character is expected.
  • Don't use the String class.

JayStel: If I comment out the delay it seems as if the Arduino locks up and then reboots. I believe this is happening because there is a test print string in the SETUP just before void loop, and it prints out again.

That's the sort of problem that could be caused by running out of memory, or by memory corruption. Using the String class is one possible cause for memory corruption. If you've made code changes and still have a problem, I suggest you post your current code.

I don't know why PeterH doesn't simply fix your code his way and see if it works. Maybe it won't? Anyhow I looked at your code and the below may have some interesting points. I put in the 2ms delay where a delay is needed in your serial byte capture loop. Why do you basically erase the String you have captured when a - is encountered?

void initMessageRead(){
  String newMsg = "";
  String str = "";
  GPRS.begin(9600);
  GPRS.println("AT+CMGR=1\r");
  delay(100);

  if( GPRS.available() ){ // if new data is coming from the GPRS UART
    while( GPRS.available() ){ // while there's still data in serial ..
      delay(2);  //delay to allow bytes to arrive in input buffer during capture loop
      char wholeMsg = (char)GPRS.read(); 
      newMsg += wholeMsg; 
      if (wholeMsg == '-') {
        newMsg = ""; // !!!!!!!what is this?

      } 
    }

  }
}

Thank You Zoomcat- Very much. I deleted all characters up to the - (minus) because I "thought" there was too much data to be buffered and, the minus symbol was the only character near the end of header that seemed consistent. Turned out that this is not needed because the buffers do in fact hold the entire received message.

Something changed last night by commenting out a delay, trying again (failed) and then removing the two slashes that made the comment. The system has worked reliably since then. I'n powered it down. Unplugged the power supply and ran I don't know how many large txt (SMS) messages to it without fail.

Not at home now, but when I get there this evening I'll post the entire code as it is loaded/running now. Plus, will reduce the delay to 2ms as you propose. I meant to do that form your previous assistance but... OOOPS....

Thank you again, You're very helpful! John