Got an Arduino Mega with 2 Serial devices connected to HardwareSerial pins:
A 10hz GPS Module(Lycos LS20031)
A Cellular Module (The ever so popular but cryptic SM5100B).
A sketch talks to both of them and sometimes works. After like 12 hours of debugging I have narrowed the problem down.
When the GPS module has a fix it screams 10hz data into Serial 3. At the same time im processing cellular AT commands in Serial2.
Occasionally Serial2 will appear to lose random characters from an AT command response. For example SOCKSTATUS has appeared as SCKSTATUS and SOCKSTATU etc. This becomes a particular problem when trying to send data. It requires an initiating command which is acknowledged by a ">" which signifies that I can start sending data. Serial2 Routinely misses this ">" which hangs my processing as my recovery commands are interpreted to be a part of the outgoing data.
I have snooped on Serial2 Using a FTDI to USB board and have verified that the cellular module does in fact send the ">" and correctly sends "SOCKSTATUS" reliably.
I noticed this problem before with SoftwareSerial and the GPS data which was fixed by increasing the SoftwareSerial buffer. Hoping this would do the same I proceded to edit HarwareSerial.cpp(57,60) to up the buffer size to:
This seems to have worsened the problem as now the module startup sequence is being missed. Reading the rest of the code I noted that this gives essentially 4 serial ports 512 bytes of memory so reduced it again to a lower value.
Is there anything I can do to increase the reliability of getting the entire message through? Or do I have to slowdown my GPS.
I am unable to see if this is a problem on the GPS module. Its data is so fast that any lost data is usually not missed by my code.
64 bytes are usually more than enough buffer for the serials. If you're getting the GPS data at 10Hz and every line is perhaps 100bytes, the processor has to process 1kB per second which is not much even for an Arduino (which is ways slower than today's PCs).
In almost every case if you loose characters on the hardware serial interfaces you made a mistake in your sketch, like having delay() calls in it. If you do your programming correctly you won't have problems loosing characters. Show us your (complete) code and we try to find the software problem.
Thanks for your responses from them I'm guessing my waitForOK is the problem. Most AT commands respond with just OK and the order of my command sequence needs to be maintained so that the module is ready for what I want to do with it. For example here I have some captured output from the snoopping wire into realterm:
I believe the whitespace line acts as a kind of checksum based entirely on the length of the previous line. But it is rather irritating as it interferes with waitForOK further.
It behaves strangely if I don't perform a small action before reading from serial3 within it. I used to have this as a Serial print which made debugging impossible. I settled for a short delay instead.
Ill post relevant areas of code but am unable to post the full code:
at_buffer is a String object
void loop () {
if(Serial3.available() >0) {
if(readATString(Serial3.read())) {
processATString();
}
}
if(Serial2.available() > 0) {
gpsLoop();
}
}
boolean readATString(char c) {
if (c == -1) {
return true;
}
if (c == '\n') {
return true;
}
if (c == '\r') {
return false;
}
at_buffer += c;
return false;
}
void waitForOK() {
//Without a println or a small delay here I never seem to get an "OK"
//Serial.println("Waiting for OK");
delay(500);
at_buffer = "";
while(Serial3.available() > 0)
{
if(readATString(Serial3.read())) {
break;
}
}
at_buffer.trim();
if(at_buffer.compareTo("OK") == 0) {
Serial.println("Found OK");
return;
} else {
if(at_buffer.length() > 2) {
Serial.println(at_buffer);
}
waitForOK();
}
}
Is it the case that ANY blocking part of code will cause this character disappearance?
Waiting half a second is NOT a small delay! With a GPS sending at 10Hz you're loosing 5 positions. This is probably not a problem but may show you that is an incredibly long time for a microcontroller.
Then you're calling your waitForOK() function recursively without limiting the recursion. This is a potential dead lock.
Another problem: You're using the String class which is badly coded for an embedded environment. The String class is designed for PC operating systems which feature Memory Management Units and much more memory (by about a factor of a million and more). It shouldn't be used in code for Arduinos (and MCU systems) that needs to run for more than a few loops (so about everything but the most simple test sketches). Forget about that class and use character arrays instead.
ahref:
I believe the whitespace line acts as a kind of checksum based entirely on the length of the previous line. But it is rather irritating as it interferes with waitForOK further.
I'm aware of the problems with the code and am constantly working to improve them. This however is not the subject of this topic so while your critique is valued and will be used to improve said code I was looking more for something that may be causing the problem other than my unoptimized code. I'll make your suggested changes and try again though
I am trying to determine what might be cause my character drop on Serial2. As explained(maybe poorly) without a delay() or a serial print waitForOK fails to do anything. It is like a shrodinger's cat problem. I do believe a 64 byte buffer is able to store an OK for half a second. I have removed the delay and will be more patient with waitForOK()(see below).
On the subject of the String object. I fail to see how its a problem granted it uses more memory but I have plenty to spare I like the functionality of String for rapid development. If memory becomes an issue I will be able to downgrade to char arrays which I hate but put up with
On the subject of waitForOK I expect OK, If i do not get it the rest of the sketch is unlikely to work. At this stage in development If i get anything other than OK im just printing it. I then workout what went wrong and rerun the code.
However due to the vanishing character issue im getting "O" and "K" or sometimes just " ".
I can remove the recursion when i reliably get either OK or CME ERROR. It is just being reset when it deadlocks as the realterm output clearly recieves OKs Netherless:
void waitForOK() {
//Without a println or a small delay here I never seem to get an "OK"
//Serial.println("Waiting for OK");
delay(5);
at_buffer = "";
while(Serial3.available() > 0)
{
if(readATString(Serial3.read())) {
break;
}
}
at_buffer.trim();
if(at_buffer.compareTo("OK") == 0) {
Serial.println("Found OK");
return;
} else {
if(at_buffer.length() > 2) {
Serial.println(at_buffer);
return;
}
waitForOK();
}
}
Should solve that problem while maintaining the behaviour i want for now. trim removes the whitespace if whats left is longer than "OK" then its likely to be a CME ERROR.
It's just a CR, isn't it?
If you look at my output image you'll see a CRLF on a seperate line after an actual command. The whitespace before this appears to match the length of the string on the line before. This does not occur with the GPS data or anything else i've seen through serial.
I understand that the output you show in the figure is from Serial2, and that the function that process it is gpsLoop(). Can you tell us what are the conditions that make gpsLoop() return? Any chance you are calling waitForOk() from gpsLoop() as well?
It's just a CR, isn't it?
Sorry I meant a LF. I believe you can control such display options in realterm.
You're using the String class, which is a very bad idea since it exposes a bug in the memory management library.
You're making recursive calls to waitForOK(). Recursion is dangerous, and certainly not needed here.
Your high level approach of reading and buffering characters received from each device and then processing them when you have a complete message is the right one to take IMO, but you have taken a different approach in waitForOK which IMO is a mistake. Rather than making this part of the sketch blocking, you would be better to use a state machine to handle the interaction with the modem.
Since you haven't posted your whole sketch it's impossible to say for certain, but from I've seen so far I suspect the lost characters are caused by your sketch waiting too long without reading the serial input, causing the internal buffers to overflow. You need to make the processing of each input stream independent and non-blocking in order to deal with all those input streams.
waitFor is not called from GPS loop. GPS loop grabs the current coords if available and chucks them into a kalman filter. As it uses tinyGPS it has something similar to my readAtString.
I believe the above may please your issues with my waitForOk. I no longer have any delays in the code.
I am unable to post my full code as it is part of a project whose Intellectual Property does not soley reside with me. I understand how this makes things difficult but I am not privy to post the exact method in which im transmitting data or storing it on removable media. I am looking more for theoretical answers if possible which largely your post is
Could you provide a reference for your String class issue. It is on the official reference documentation and as such I thought it would be ok to use. With your reference I'll be able to evaluate things and hop to char arrays ok?
I understand that you guys all get silly questions all day but could you please try to not come across as hostile. Being firm is fine, i've already re-written waitForOK twice simply from being embarrassed at everyone pointing out it's flaws. Spatula's approach is fine PeterH I found you slightly rude. Sorry :(. I understand if that last sentence might piss you off and you might not want to continue helping. I am however grateful that you are providing new insight the state machine idea is fantastic.
Just so I understand the concept is this approach the sort of thing you had in mind:
Send a command that needs an OK
Switch to a state that solely listens for an OK and doesn't process anything else
Receive an OK and go back to the PROCESS ANYTHING state.
The problem with that is how to handle sequencing. Say the following needs to happen:
A
OK
B
OK
C
OK
When in the OK state and i get OK how do I know which command to send next? What comes to mind is a FIFO data structure? Would this work?
I link the discussion and not the material about the String issues because poor Nick Gammon is spending a lot of time advising people not to use the String class. I think Pylon should be thanked not only for his advice, but also for sparing Nick the hassle to come here and explain the issue one more time.
As for states, if I understand your flow you are not waiting for commands that may branch into different states but for answers that can trigger a change to the next state. This may help you design a state diagram that is roughly sequential. As I see it, OK is more a triggering condition than a state, and you shouldn't conflate your states (A, B, C) with the (sub)states you just need to process the serial input.
In other words, you start from state A, initialize your settings and call something like waitOk, which manages its own state. Possibly, waitOk should know nothing about A, B, C. When waitOk returns you are still in state A, you do whatever you need to do to process the answer, and move to state B.
