Unable to Receive More Than 64 Bytes of Data via Serial.read()

I have a little project of my own to receive SMS messages with Arduino UNO by sending AT commands to GSM module via serial communication.

I don’t have the Arduino UNO or GSM module yet, so I’m simulating Arduino code with CodeBlocks, send AT commands to a pair of virtual serial port created with a null-modem emulator. In order to test my logic, I send response from other end of virtual serial port through a serial terminal program (Realterm for my case) to Arduino as if a GSM module would respond to AT commands.

I have Arduino send AT+CMGR=1 to serial port in order to read SMS with index 1. GSM module would send back something like below.

+CMGR: “REC UNREAD”,"+XXXXXXXXXXXXX", “”,“02/01/30,20:40:31+00”
Test SMS Message.
OK

So I send above data from my end of virtual serial port to simulate GSM module. But the problem is that Arduino only receives +CMGR: “REC UNREAD”,"+XXXXXXXXXXXXX", “”,“02/01/30,20:40:31+00” which is 63 bytes of data. When I google about this problem, I saw that Arduino is limited to a 64 byte serial buffer.

What am i doing wrong? Is it possible to read more than 64 bytes through serial port without changing HardwareSerial.h in core library? I’m sharing the code I use for reading serial port at the end of the post.

I’m not a native English speaker, I hope I was able to describe the problem and my setup clearly.

Thanks in advance.

#include <Arduino.h>
const byte numChars = 128;
char receivedChars[numChars];
boolean newData = false;

void setup() {
    Serial.begin(9600);
    Serial1.begin(9600,5);
}

void recvWithEndMarker() {
    static byte ndx = 0;
    char rc;
    delay(1000);
    while (Serial1.available() > 0 && newData == false) {

        rc = Serial1.read();

        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
            ndx = numChars - 1;
        }

    }
    if (ndx>0){
        Serial.println(ndx);
        receivedChars[ndx] = '\0';
        newData = true;
        ndx = 0;
    }

    if (newData){
        Serial.println("RECEIVED DATA");
    }
}

void showNewData() {
    if (newData == true) {
        Serial.print("This just in ... ");
        Serial.println(receivedChars);
        newData = false;
    }
}


void loop() {
    recvWithEndMarker();
    showNewData();
}

Is it possible to read more than 64 bytes through serial port without changing HardwareSerial.h in core library?

Yes, of course.

Robin's examples in Serial input basics - updated, which you have obviously seen, allow you do do this because each character is dealt with as it arrives thus the buffer never gets full.

Yeah I've seen that. I used same exact code yet I fail to receive more than 64 bytes.

Why does your recvWithEndMarker() function never check for the receipt of an end marker ?

Your problem is this:

delay(1000);

You are receiving data at up to ~1000 chars/second, but you spend a full second, doing absolutely NOTHING. If a message >64 bytes arrives during that second, everything beyond the 64th character will be discarded, because the buffer is full.

If you want to receive large messages, you MUST check, and empty, the buffer often enough that it can NEVER overflow. That means you MUST check it AT LEAST every 64 mSec.

Regards,
Ray L.

The Serial Input Buffer has space for 64 bytes so if you need to receive a longer message your program must remove bytes from the buffer (with Serial.read() ) before it gets full.

You seem to have added code to my recvWithEndMarker() function which should not be in it and removed code that should be in it.

Try working with my version.

If that does not work then please post an example of the data that is being sent. And be sure to make visible to us all non-printing characters that may be in the message.

...R

Serial.begin(9600); Simple rule-of-thumb: One character at that speed takes a little over 1ms to be sent.

That's an awfully long time, even for an eight bit micro running at a glacial 16MHz.

Robin2:
The Serial Input Buffer has space for 64 bytes so if you need to receive a longer message your program must remove bytes from the buffer (with Serial.read() ) before it gets full.

You seem to have added code to my recvWithEndMarker() function which should not be in it and removed code that should be in it.

Try working with my version.

If that does not work then please post an example of the data that is being sent. And be sure to make visible to us all non-printing characters that may be in the message.

...R

I've tried your version, it worked. But the response would arrive in three pieces like so below.

This just in ... +CMGR: "REC UNREAD","+XXXXXXXXXXXXX", "","02/01/30,20:40:31+00"
This just in ... Test SMS Message.
This just in ... OK

In order to concatenate 3 pieces data into 1, I have to be able to understand when serial port finally stops receiving data. How can I do that? Clearly I cannot trust Serial.available(), since after the first piece of data, rest of data keeps coming after printing the first part of data.

In my experience, Serial.available is completely trustworthy.

AWOL:
In my experience, Serial.available is completely trustworthy.

I thought so too. But I send the data with a text file as a whole from the terminal. Yet I receive it in 3 pieces which indicates that after detecting the end marker '\n', next part of the data keep flowing. How will I understand when the transmission really ends?

Maybe it's all caused by the simulated environment. Maybe I should just buy the real equipment, and try on it. But I feel like I'm doing something wrong here and I'm at a loss to find what it is.

Not sure if it is the correct way of working with serial ports but I’ve found a solution to my problem. When I receive a data from serial port, I checked the port if there is more incoming data using Serial.peek() for a specified time. After this specified time passed, If there is data in the port, I add the data into my buffer, if there is no data then I stop listening and print the data. Below is the code for this approach. Thank you all for your replies.

#include <Arduino.h>
const byte numChars = 250;
char receivedChars[numChars]; // an array to store the received data
const unsigned long setTime = 5;
unsigned long startTime;
unsigned long endTime;
unsigned long passedTime;
boolean newData = false;

void setup() {
 Serial.begin(9600);
 Serial1.begin(9600,5);
 Serial.println("<Arduino is ready>");
}

/*void FillBuffer(){
            receivedChars[ndx] = rc;
            ndx++;
            if (ndx >= numChars) {
                ndx = numChars - 1;
            }
}*/

void recvWithEndMarker() {
    char rc;
    static byte ndx = 0;

    while (Serial1.available() > 0 && newData == false) {
        startTime = millis();
        rc = Serial1.read();
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
            ndx = numChars - 1;
        }
    }

    if (Serial1.peek()==-1 & startTime>0){
        endTime=millis();
        passedTime=endTime-startTime;
        if (passedTime>setTime){
            receivedChars[ndx] = '\0'; // terminate the string
            ndx = 0;
            newData = true;
            startTime=0;
            endTime=0;
            passedTime=0;
        }
    }

}

void showNewData() {
 if (newData == true) {
 Serial.print("This just in ... ");
 String message;
 message = receivedChars;
 Serial.println(message);
 Serial.println(message.length());
 newData = false;
 }
}

void loop() {
 recvWithEndMarker();
 showNewData();
}

When I receive a data from serial port, I checked the port if there is more incoming data using Serial.peek() for a specified time.

Why not use Serial.available() rather than Serial.peek() ?
It is exactly what it is designed for.

UKHeliBob:
Why not use Serial.available() rather than Serial.peek() ?
It is exactly what it is designed for.

Tride using available() instead of peek(), it worked too. Thanks for pointing it out.:slight_smile:

Cauchy9:
I thought so too. But I send the data with a text file as a whole from the terminal. Yet I receive it in 3 pieces which indicates that after detecting the end marker '\n', next part of the data keep flowing. How will I understand when the transmission really ends?

The reality is that using '\n' as the end-marker the data is being sent in three parts and my code correctly receives the data.

Maybe you need a different end-marker. How do you (as a human) identify when the message ends?

For example if the end of the data is always signalled by "OK" and if a 'K' never appears anywhere else in the data you could set 'K' as the end marker.

If you can't reliably use 'K' as an end marker then you would probably need to make a substantial change to the code to check for (say) "OK" as the indicator of the end of the message.

To do that you could use 'O' as the initial marker and when you get an 'O' read another character. If it is a 'K' it means the end. If it is not then add both the 'O' and the other character into the received data and go back to looking for an 'O'

...R