Software Serial charactor limit using readString (????)

Projects Networking, Protocols, and Devices
1

I'm pulling my hair out! I have been trying to pull strings over to the arduino via blue tooth and seem to be hitting a brick wall at 44 characters. Some of the strings I am trying to pull over are almost a 100 characters, but no matter what I send, I get either 44 characters or the number of characters sent, which ever is least.

I am using the arduino mini pro, softwareSerial and sending from either my phone or a tablet to a blue tooth module. I have tried both devices to send the data, two different blue tooth transceivers on the arduino, always with the same result....send 35 characters and I receive 35 characters....send more than 44 characters and I receive only 44 characters.

Is there some limiting factor related to SoftwareSerial? The basic code I am using to receive this data is ...

#include <SoftwareSerial.h>
SoftwareSerial mySerial(2, 3); // RX, TX
mySerial.begin(9600);

String msg;
msg.reserve(200);

if (mySerial.available()) {
       msg = mySerial.readString();
       Serial.print("bytes read: "); Serial.println(msg.length());
       }
2

ADDITIONAL INFO

I have figured out that this does not happen until my used code increases to somewhere over 11,0000 bytes. When I commented out routines (unrelated to the receiving of data) until the code was below 9,000 bytes, a string of 90 characters was being received as all 90 characters. As I added those routines back in and the code space usage went above 11,000 bytes 90 characters where sent and only 76 characters where received. When I approached 20,000 bytes of code space used, I only received 44 bytes of the 90 sent. Something is being stepped on and I don't know how to prevent it. Obviously, the reserve() method is not preventing it. Since the max code space is 30,720, why would this start to happen when I am not that close to the max.

Also I found that it really didn't matter what code I added....It only mattered how much code space was used regardless of what used it.

3

We probably need to see the entire program.

msg.reserve only sets the initial allocation of the String. If it needs to grow (which I can't tell from your excerpt), it will reallocate more. Maybe too much.

In general, String is bad juju. Over long periods of time, it can cause random failures. Just declare a byte array and store characters into it. See Serial Input Basics.

Cheers,
/dev

4

Since the max code space is 30,720, why would this start to happen when I am not that close to the max.

Well there is a difference in program memory space and the 2k memory space for dynamic variables and static text strings like "bytes read: ".

5

Switching to a char array did not help.

I have a reduced code to a bare minimum that shows the problem. I am sending several messages via blue tooth, but I get different results depending on how much code space I have used up. (the tilde is just a reference character)

The messages sent are....

"In our world, you were either a bully, a toady, or one of the nameless rabble of victims!~",
"Some men are Baptists, others Catholics. My father was an Oldsmobile man.~",
"Eat, drink and be merry for tomorrow you may diet~",
"Seasons Greatings~"

the first three seem to get truncated at 67 or 68 bytes, but the shorter once go through.

Minimal code but still doesn't work correctly

#include <SoftwareSerial.h>

SoftwareSerial mySerial(2, 3); // RX, TX

char aMessage[200];                               // to hold the message received
char cha;                                         // for reading char one at a time
byte messageSize;                                 // to hold the received message sixe

void setup() {
          
          
          Serial.begin(9600);                     // start the standard serial port for debugging 
          Serial.println("regular serial ok");    // just to make sure it's open
          
          mySerial.begin(9600);                   // openning the second serial port for receiving data.
          } // setup

void loop() {

          if (mySerial.available()) {
                    for (int c = 0; c <=99;c++) aMessage[c]=0;        // clear aMessage in prep for new message
                    messageSize = 0;                                  // set message size to zero

                    while (mySerial.available()) {                    // loop through while data is available
                              cha = mySerial.read();                  // get character
                              aMessage[messageSize]=cha;              // append to aMessage
                              messageSize++;                          // bump message size
                              delay(20);                              // just to slow the reads down a bit
                              } // while
 
                    aMessage[messageSize]='\0';                       // set last character to a null
                    Serial.print(messageSize);                        // print the message size
                    Serial.print(' ');
                    Serial.println(aMessage);                         // and then the message                              

                    } // if available
        } // loop

And this is the output I'm getting...

regular serial ok
68 In our world, you were either a bully, a toady, or one of the namel 
67 Some men are Baptists, others Catholics. My father was an Oldsmobil
50 Eat, drink and be merry for tomorrow you may diet~
18 Seasons Greetings~

The leading number is the length of the array received. What am I doing wrong???

6

You're incrementing your index messageSize without doing bounds checking which could result in clobbering memory outside of the range of your array.

Adding a 20ms delay in your read loop isn't a good idea. Look for the terminating character on each string coupled with either a timeout or some way to detect a new string starting.

You don't need to clear your buffer (you're not really doing it anyway). The string terminating character is sufficient.

Sometimes it helps to have the output baud rate greater than the input baud rate to avoid buffer overflow. E.G., Serial.begin(115200).

Also, SoftwareSerial can cause headaches sometimes. Use NeoSWSerial (look on github).

7

Like jboyton said, the delay is a problem. Did you see any delays in Serial Input Basics?

Here's what's happening:

  1. Serial.available() == 0, hundreds of times, until the first character starts
  2. an interrupt occurs and doesn't return until the entire character
    is received and stored in the input buffer. This takes about 1ms.
  3. Serial.available() == 1
  4. Serial.read() returns the 1st character
  5. 1st character is stored in aMessage
  6. delay(20) starts, but it's really just a loop like this:
    uint32_t start = millis();
    while (millis() - start < 20)
       ; // do nothing but loop...
  1. after about 1ms (still in delay loop), an interrupt occurs and
    doesn't return until the entire character is received and
    stored in the input buffer. This takes about 1ms (t = 2ms).
    8-27) 19 more characters are received and stored (t ~= 21ms).
  2. the delay loop finishes (and your loop() gets called again)
  3. Serial.available() == 20
  4. Serial.read() returns the 2nd character
  5. 2nd character is stored in aMessage
  6. delay(20) starts
    33-53) 20 more characters are received and stored (t ~= 41ms)
  7. the delay loop finishes
  8. Serial.available() == 40
  9. Serial.read() returns the 3rd character
  10. 3rd character is stored in aMessage
  11. delay(20) starts
    59-79) 20 more characters are received and stored (t ~= 61ms)
  12. the delay loop finishes
  13. Serial.available() == 60
  14. Serial.read() returns the 4th character
  15. 4th character is stored in aMessage
  16. delay(20) starts
    85-86) 20 more characters are received and stored (t ~= 81ms). But the input buffer is only 64 characters long, so the last 16 characters are thrown away.
  17. the delay loop finishes
  18. Serial.available() == 64
  19. Serial.read() returns the 5th character
  20. 5th character is stored in aMessage
  21. delay(20) starts
    101-121) 20 more characters are received (t ~= 101ms). But the input buffer has room for just one, so the last 19 characters are thrown away.
  22. the delay loop finishes
  23. Serial.available() == 64
  24. Serial.read() returns the 6th character
  25. 6th character is stored in aMessage
  26. delay(20) starts
    127-138) 11 more characters are received (t ~= 112ms). But the input buffer has room for just one, so the last 10 characters are thrown away.
  27. the delay loop finishes
  28. Serial.available() == 64
  29. Serial.read() returns the 7th character
  30. 7th character is stored in aMessage
  31. delay(20) starts
    -- no characters received, as the first sentence is 91 characters long --
  32. the delay loop finishes
  33. Serial.available() == 63
  34. Serial.read() returns the 8th character
  35. 8th character is stored in aMessage
  36. delay(20) starts
    goto 144 for the remaining 62 characters, for a total of 70 characters

delay is rarely a good thing to do. Interrupts still happen in the background, and the SoftwareSerial input buffer overflows because the delay(20) lets 20 characters stack up, unhandled.

The referenced tutorial is good stuff. It even includes an example for a terminating character like your '~' (aka "end marker). And follow up on jboyton's other suggestions

Cheers,
/dev

8

Holly mackerel...
I don't know how you were able to put those responses together that fast, but thank you very much. I will dig in and start trying to apply what you've both given me.

The delay was something I tried early on and never removed. I noticed that if I put a Serial.println in while debugging, i seamed to get more characters, so I thought slowing it down a bit might help. It didn't, of course. I will dump it.

I did go over the basics link you sent, but clearly not well enough to apply it very well.

Charlie

9

Well, you guys got me there. I dropped the delay and used the recvWithEndMarker() function pretty much without out any changes (except to a couple variable names). It seems to be working correctly now. There is really a lot about how the Serial data is sent and received that I need to understand better, but at least my feet are pointed in the right direction now.

I have not replaced the SoftwareSerial library with NeoSWSerial yet, but that will be the next refinement.

I will fold this back into the regular program tomorrow and I anticipate it working exactly as it does now. Also I need to get better acquainted with the functions for manipulating char arrays. Again, tomorrow.

Thank you very much.

10

I can guess what the recvWithEndMarker() function does but where is it? Is it in the core or some library? I'm just curious.

Depending on what you're doing SoftwareSerial may be just fine. But timing issues can expose its weaknesses. It's basically a bandwidth hog (about 95% at 9600 baud and 16MHz clock). Also, I should have mentioned that the current implementation of NeoSWSerial only works for ASCII data.

Don't feel bad about being confused. Serial is inherently confusing because timing is important. I couldn't look at your script and figure out exactly what was wrong, only that several things appeared problematic. It seems like /dev saw it all in one glance, as if he were in the UART looking out. But that's not normal.

11

jboyton: In message #2 above, /dev gave a link to a nice little tutorial on reading the serial data. In one of his example code segments, he shows the recvWithEndMarker() function. It pretty much worked "out of the box" for me and gave me a much better understanding of what I needed to know about reading serial data. (although I am still at cadet status).

12

(deleted)