Large serial read into array problems

Projects General Guidance
1

Hi,

Currently I am undertaking a project that involves texting the unit then returning the GPS coordinates to the number that texted. Using the Arduino UNO and a GPS/GSM Shield (SIM908 based).

The shield interacts using AT commands. I have set the arduino as a serial interface and have been able to retrieve a text, open it and then obtain coordinates. This has worked successfully and I have attached a screenshot of this working.

However I would like the arduino to effectively automate this. I have got it to the stage where it can detect an incoming sms, once this is done an AT command is send to the shield that returns the coordinates. I am attempting to enter the coordinates into a char array before sending, this seems to be limited to so many characters before stopping. I have a feeling this has something to do with the serial buffer size.

Entering GPS result into array

Serial.println("AT+CGPSINF=32");//this outputs the gps into the serial 
delay(500);
 
 while (Serial.available() > 0) 
                                   
    {
        for(i=0;i<100;i++ ) 
        {
            sr = Serial.read(); // Read a character
            GPS[i] = sr; // Store it

        }
    }

This is how I manually control it when the arduino is set as a serial interface.

AT

OK
AT+CGPSPWR = 1

OK
AT+CGPSRST = 1

OK
AT+CMGD = 1,4

OK

+CMTI: "SM",1
AT+CMGR= 1

+CMGR: "REC UNREAD","+447540310540","","17/01/26,09:46:02+00"
Test

OK
AT+CGPSINF=32

32,092545.000,V,0.000000,N,0.000000,E,0.00,0.00,011110,,E,A
OK

The attached file is a screenshot of what the current array is saving once printed back.

Thanks everyone, as I say, I think the problem is to do with the serial buffer so if anyone has any idea why only half of the serial is getting added to the array?

Screen Shot 2017-01-26 at 09.53.27.png
Screen Shot 2017-01-26 at 09.53.27.png930×772 80.9 KB

2

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data. The examples use an array of 32 bytes but you can change that by changing the value of the constant numChars

...R

3

Your code is assuming that just because Serial.available() returns something that is not zero then everything is going to be there. You also assume that everything means 100 characters. It's possible that you are stopping reading while the device is still sending.

You should rewrite this to keep reading until you see the terminating string "OK", and don't increment your buffer pointer unless you actually read something. To prevent an infinite loop, you should also have a timeout so that you don't wait forever in case something goes wrong.

Put all of the above logic into a utility method so that you can re-use it with different AT commands.

Also clear the serial buffer before you execute the AT command to get rid of anything that might be left over from previous commands.

4

davidrh:
Your code is assuming that just because Serial.available() returns something that is not zero then everything is going to be there.

It most certainly does not. You seem to have missed the whole point of why I wrote it.

What it does do in any iteration is collect all the characters that have already arrived.

It may need several iterations of loop() before it gets to the end-marker for a particular message.

...R

5

Sorry @Robin2 - I was referrring to the OP's code. To quote your linked post:

And because data arrives relatively slowly it is easy for the Arduino to empty the serial input buffer even though all of the data has not yet arrived. Many newcomers make the mistake of assuming that something like while (Serial.available() > 0) { will pick up all the data that is sent. But it is far more likely that the WHILE will empty the buffer even though only part of the data has arrived.

This sounds like what the OP is seeing. The delay(500) may give enough time for the response to be received, but it may also allow the serial buffer to overflow if the response is too large. From memory, the buffer size is hard coded in the serial library.

6

davidrh:
Sorry @Robin2 - I was referrring to the OP's code.

Humble apologies. I mistook Reply #2 as a response from the OP.

A trip to the Opticians may be in order? or the psychiatrist?

...R

7

@

davidrh:
Sorry @Robin2 - I was referrring to the OP's code. To quote your linked post:

This sounds like what the OP is seeing. The delay(500) may give enough time for the response to be received, but it may also allow the serial buffer to overflow if the response is too large. From memory, the buffer size is hard coded in the serial library.

I thought that, I have tried a range of delays from no delay at all up to delay(5000) and still getting the same result.

8

james490:
I thought that, I have tried a range of delays from no delay at all up to delay(5000) and still getting the same result.

The examples in my link don't need any delay() and don't miss data either.

...R

9

By default the arduino only has (from memory), a 64 byte buffer and can easily get overrun , if you need to change this you have to make a copy of the board definition file and then edit the definition

I had a similar problem when handling incoming sms messages , it all worked fine with short messages but longer ones got truncated

10

sureview:
By default the arduino only has (from memory), a 64 byte buffer and can easily get overrun

The code in my examples empties the Serial input buffer before there is any chance of it overflowing.

...R

11

I thought that, I have tried a range of delays from no delay at all up to delay(5000) and still getting the same result.

Even removing the delay won't make it work as you have other problems as previously highlighted. Have a look at the code @Robin2 linked to as a better example of how to read from the serial buffer. What baud rate are you using? If you do it properly I think the Arduino is supposed to be good for up to 57,600 with software serial.

Even if the problem ends up lying elsewhere, starting with solid code won't do any harm and will probably solve the next problem that you haven't had yet.

12

sureview:
By default the arduino only has (from memory), a 64 byte buffer and can easily get overrun , if you need to change this you have to make a copy of the board definition file and then edit the definition

I had a similar problem when handling incoming sms messages , it all worked fine with short messages but longer ones got truncated

Sureview - I can't find any location on my Mac with the board definition file softwareserial.h in order to increase the buffer size. Can you help me?

Thanks a lot!
Best

John

13

It will not be in a boards file. It will be in the library files.

But don't go that route; base your code on Robin's examples.

14

sterretje:
It will not be in a boards file. It will be in the library files.

But don't go that route; base your code on Robin's examples.

Ok, will check Robis examples.
But just for the sake of interest - I can find the Serialsoft.cpp or the corresponding *.h file in my system. Not in the libraries. Is this somewhere hidden?

Like sureview wrote I have the same issue: my Sim8000L module sends a SMS with more than 64 bytes. MAybe 128 bytes. I would just like to go the simple route for now with buffering it as I'm not experienced too much with Arduino yet..

Thanks for any hint to temporary increase the buffer size to 128bytes.

John

15

It's a ling time since I looked at this problem , I am sure it was in the boards definition,but I could be wrong, its not I the library I am sure as its part or the core, I found it by a google search , so give that a try, I would also try the search on this forum, its definitely documenter some ware

I am sire its in the boards config as the article I read suggested you make a copy of the boards definition and rename it so you could have a standard and modified version.....

16

Johncoffee:
I would just like to go the simple route for now with buffering it as I'm not experienced too much with Arduino yet..

Using the examples in Serial Input Basics will be a lot simpler than fiddling around with the SoftwareSerial code.

And even after you have fiddled with SoftwareSerial you will still need code like in my examples to get the data out of the input buffer.

...R

17

Johncoffee:
I would just like to go the simple route for now with buffering it as I'm not experienced too much with Arduino yet..

It will not solve problem that your code is just unreliable; davidrh already pointed it out in reply #2.

Again, go Robin's route. It's reasonably easy to understand and you can ask if you don't understand.