Limit on serial buffer

Projects General Guidance
1

Greetings!

I have been working on Nordic NRF52840 USB dongle and after trying couple of things I decided to give up and use it just as BLE UART device which sends the data it receives from mobile app to arduino mega on UART at 115200 baudrate. My friend who made this app and me mutually agreed to send this data from app in a format prescribed like <1,2,3,6,4,7,0,asf,23,hg> likewise I took this idea from the arduino example for serial communication basics to avoid any data loss. Everything was working fine until we were required to send an image and audio file.

The app sends Image as base 64 with a really long string of data same is the case for audiio. When we tried to send this to the NRF52840 connected to CP2102 module we could see all those base 64 stuff being received on to the tag in serial monitor, so we later decided to connect it to arduino mega so that it can be stored in SD card. We also kept the serial monitor commands which helps us to view whats exactly happening there. The problem is that Arduino doesn't receive complete string for image and neither for audio. SO my question is there any limit on how long string of data I can receive over UART? Im sharing the code that I use on arduino Mega

/*
 * This example accepts data from nrf52840 and sends it to SD card
 */

#include <SoftwareSerial.h>
SoftwareSerial mySerial(10, 11); // RX, TX

#include <SPI.h>
#include <SD.h>

File myFile;

const byte numChars = 100;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing

boolean newData = false;

//============
void setup() {
    Serial.begin(9600);
    mySerial.begin(115200);
    Serial.println("Demonstrate data parsing received from mobile app");
    Serial.println();
    if (!SD.begin(53))
    {
    Serial.println("initialization failed!");
    while (1);
   }
  Serial.println("initialization done.");
  delay(5000);
    GetDataFromSDCard();
}

//============

void loop() 
{
  
    recvWithStartEndMarkers();
    if (newData == true) 
    {
     strcpy(tempChars, receivedChars);
     SendTOSDCard();
     newData = false;
    }
}

//============

void recvWithStartEndMarkers() 
{
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (mySerial.available() > 0 && newData == false) {
        rc = mySerial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

//============
void SendTOSDCard()
{
    myFile = SD.open("test.txt", FILE_WRITE);
  if (myFile) 
  {
    Serial.println("Writing to test1.txt...");
    Serial.println(tempChars);
    myFile.println(tempChars);
    myFile.close();
    Serial.println("done.");
  } 
  else
  {
    Serial.println("error opening test.txt");
  }
}

//============
void GetDataFromSDCard()
{
  delay(5000);
  myFile = SD.open("test.txt");
  if (myFile) {
    Serial.println("test.txt:");

    // read from the file until there's nothing else in it:
    while (myFile.available())
    {
      mySerial.print("{<");
      mySerial.write(myFile.read());
      mySerial.print(">}");
    }
    // close the file:
    myFile.close();
  } else {
    // if the file didn't open, print an error:
    mySerial.println("{<1,0,,,,,,,,,,,,,,,,,>}");
  }
}

this is the output I got in serial Monitor

21:26:26.412 -> Demonstrate data parsing received from mobile app
21:26:26.479 -> 
21:26:26.479 -> initialization done.
21:27:08.163 -> Writing to test1.txt...
21:27:08.198 -> 1,0,Sharma,Shruti,Anuja,5556,fgh,F,16,09,1998,0,bomber,11,03,2020,21,27,07
21:27:08.267 -> done.
21:27:47.004 -> Writing to test1.txt...
21:27:47.038 -> 3,1,Laceration,0,Fall,11,03,2020,21,27,45
21:27:47.073 -> done.
21:27:55.421 -> Writing to test1.txt...
21:27:55.421 -> 4,1,0,11,03,2020,21,27,54
21:27:55.454 -> done.
21:28:08.460 -> Writing to test1.txt...
21:28:08.495 -> 5,1,0,5566,11,03,2020,21,28,07
21:28:08.530 -> done.
21:28:20.526 -> Writing to test1.txt...
21:28:20.560 -> 6,1,0,Intact,5,11,03,2020,21,28,19
21:28:20.595 -> done.
21:28:33.278 -> Writing to test1.txt...
21:28:33.325 -> 7,1,Analgesics,Fentanyl,55669,0,11,03,2020,21,28,32
21:28:33.378 -> done.
21:28:44.221 -> Writing to test1.txt...
21:28:44.275 -> 8,1,0,56909,0,11,03,2020,21,28,43
21:28:44.322 -> done.
21:29:03.367 -> Writing to test1.txt...
21:29:03.402 -> 9,1,hello there, it's testing 1,2,3,11,03,2020,21,29,02
21:29:03.437 -> done.
21:33:21.767 -> Writing to test1.txt...
21:33:21.767 -> 12,1,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDACgcHiMeGSgjISMtKygwPGRBPDc3PHtYXUlkkYCZlo+A
21:33:21.887 -> jIqgtObDoKrarYqMy
21:33:21.887 -> done.
21:34:18.303 -> Writing to test1.txt...
21:34:18.338 -> 13,1,AUDIO_20200311_213348_.3gp,AAAAGGZ0eXAzZ3A0AAAAAGlzb20zZ3A0AAADgG1vb3YAAABsbXZoZAAAAADajrxs2o6
21:34:18.439 -> done.

what we noticed is after the last line appears on serial monitor the app exits itself and the BLE disconnects from the app. But Im pretty sure that my mega does not reset to make this happen. what I probably think of is I need to increase the number defined for numChars which is currently 100, but I'm not quite sure about it as well. If it had to be increased what should be the number what is the maximum number it can hold? the image is supposed to be of nearly 200Kbs.

What should be done? any suggestions?

Thanks in advance

2

SoftwareSerial does not work at 115200. Maximum according to those that know is 38400.

3

SO my question is there any limit on how long string of data I can receive over UART?

Not withstanding what ieee488 has correctly told you, the answer to that question is no, there is no limit provided your code can empty the buffer fast enough to stop it overflowing. At 115200 baud your software must be able to deal with something like 11500 bytes in 1 second.

4

ieee488:
SoftwareSerial does not work at 115200. Maximum according to those that know is 38400.

it works fine on software serial tho. Im not having any problems with it till now.

PerryBebbington:
Not withstanding what ieee488 has correctly told you, the answer to that question is no, there is no limit provided your code can empty the buffer fast enough to stop it overflowing. At 115200 baud your software must be able to deal with something like 11500 bytes in 1 second.

Thanks for reply, then why I can't get the entire string of the image and audio?

5

Count:
Thanks for reply, then why I can't get the entire string of the image and audio?

Three things ...

Simplify the program by taking out the stuff that writes to the SD Card and just check that you can receive the data.

I doubt very much that audio data can fit completely in 100 bytes - which is all your program is designed to receive. If the program sending the data is breaking it up into 100 byte chunks then change it so the chunks are 62 bytes (which will be 64 when the start and end markers are added) so each complete message will fit into the serial input buffer.

Audio and video data is usually binary data so the values 60 and 62 (< and >) are likely to be part of the data which will confuse your current Arduino program. My Tutorial was written to receive text rather than binary data.

..R

6

Count:
it works fine on software serial tho. Im not having any problems with it till now.

So now its not working.

SoftwareSerial uses heaps more processing power than hardware serial.

Try switching to one of the 3 other hardware serial ports on the Mega and if the problem changes.

7

Robin2:
Three things ...

Simplify the program by taking out the stuff that writes to the SD Card and just check that you can receive the data.

I doubt very much that audio data can fit completely in 100 bytes - which is all your program is designed to receive. If the program sending the data is breaking it up into 100 byte chunks then change it so the chunks are 62 bytes (which will be 64 when the start and end markers are added) so each complete message will fit into the serial input buffer.

Audio and video data is usually binary data so the values 60 and 62 (< and >) are likely to be part of the data which will confuse your current Arduino program. My Tutorial was written to receive text rather than binary data.

..R

Ok I will check into that. Can you share me the link to your tutorial Please?

srnet:
So now its not working.

SoftwareSerial uses heaps more processing power than hardware serial.

Try switching to one of the 3 other hardware serial ports on the Mega and if the problem changes.

That's not the problem in issue here. Software serial is working perfectly fine. The entire thing will be trnasfered on hard wired UART of crystal-less SMD 328. It is necessary for formfactor issues and other things

8

Ok well the problem is the data for image and audio is not being completely received it can be seen from the output below:

Code in Use

/*
 * This example accepts data from nrf52840 and sends it to SD card
 */

#include <SoftwareSerial.h>
SoftwareSerial mySerial(10, 11); // RX, TX

#include <SPI.h>
#include <SD.h>

File myFile;

const byte numChars = 100;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing

boolean newData = false;

//============
void setup() {
    Serial.begin(9600);
    mySerial.begin(115200);
    Serial.println("Demonstrate data parsing received from mobile app");
    Serial.println();
    if (!SD.begin(53))
    {
    Serial.println("initialization failed!");
    while (1);
   }
  Serial.println("initialization done.");
  delay(5000);
    GetDataFromSDCard();
}

//============

void loop() 
{
  
    recvWithStartEndMarkers();
    if (newData == true) 
    {
     strcpy(tempChars, receivedChars);
     SendTOSDCard();
     newData = false;
    }
}

//============

void recvWithStartEndMarkers() 
{
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (mySerial.available() > 0 && newData == false) {
        rc = mySerial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

//============
void SendTOSDCard()
{
  Serial.println(tempChars);
  
  /*
    myFile = SD.open("test.txt", FILE_WRITE);
  if (myFile) 
  {
    Serial.println("Writing to test1.txt...");
    Serial.println(tempChars);
    myFile.println(tempChars);
    myFile.close();
    Serial.println("done.");
  } 
  else
  {
    Serial.println("error opening test.txt");
  }*/
}

//============
void GetDataFromSDCard()
{
  delay(5000);
  myFile = SD.open("test.txt");
  if (myFile) {
    Serial.println("test.txt:");

    // read from the file until there's nothing else in it:
    while (myFile.available())
    {
      mySerial.print("{<");
      mySerial.write(myFile.read());
      mySerial.print(">}");
    }
    // close the file:
    myFile.close();
  } else {
    // if the file didn't open, print an error:
    mySerial.println("{<1,0,,,,,,,,,,,,,,,,,>}");
  }
}

This is what I got when I connected BLE to arduino and looked for the output on serial monitor

09:42:06.091 -> 12,1,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDACgcHiMeGSgjISMtKygwPGRBPDc3PHtYXUlkkYCZlo+A
09:42:06.184 -> jIqgtObDoKrarYqMy

So I decided to directly connect my BLE to Cp2102 module and see on serial monitor what is gives for an image. This what I got in that case
See attached file becoz it exceeds the maximum words limit of post

That's whole LOT OF DATA!!

clearly Im not receiving it completely. I read about the byte that it only contain 0-255 if that's the case who can I receive all of that? I'm really confused

Output for image.txt (328 KB)

9

Count:
Ok I will check into that. Can you share me the link to your tutorial Please?

Apologies, as you are using an example from it, I assumed you were already were familiar with it.

Serial Input Basics

...R

10

Count:
I read about the byte that it only contain 0-255 if that's the case who can I receive all of that? I'm really confused

A single byte by itself holds 0 to 255 as you say. Multiple bytes are combined together to hold bigger numbers. Serial transmits 1 byte at a time, well, perhaps more accurately, 1 bit at a time but in blocks of 8 bits, making 1 byte at a time. For data types that use more than 1 byte they are split into individual bytes, transmitted then re-assembled at the receiver into the appropriate data type.

You might find Arduino reference helpful, scroll down to Variables and below that Data Types.

11

PerryBebbington:
A single byte by itself holds 0 to 255 as you say. Multiple bytes are combined together to hold bigger numbers. Serial transmits 1 byte at a time, well, perhaps more accurately, 1 bit at a time but in blocks of 8 bits, making 1 byte at a time. For data types that use more than 1 byte they are split into individual bytes, transmitted then re-assembled at the receiver into the appropriate data type.

You might find Arduino reference helpful, scroll down to Variables and below that Data Types.

Yes I read into that, What Im not getting here is if you see the image is represented by big data how to do I re assemble it arduino end if I have t get it broken into 255 parts? do I need to append or something?

12

Count:
how to do I re assemble it arduino end if I have t get it broken into 255 parts? do I need to append or something?

Images are not broken into 255 parts.

The fundamental unit of data in a PC is a byte which can hold any value from 0 to 255. Larger numbers must be made up from a combination of 2 or more bytes. For example on an Uno or Mega an int (integer) is made from 2 bytes and can hold a value from -32767 to + 32766.

Most colour photos are made up from thousands of pixels with each pixel represented by 3 bytes, one each for the quantity of Red Green and Blue at that point in the image. A black and white image could be represented by a single byte per pixel representing the shade of Grey.

Because photos have thousands of pixels they are usually far too big to fit in the limited memory of an Uno or Mega - for example a very basic 160x120 pixel image will have 19,200 pixels and a minimum of 19200 bytes.

Most cameras compress the image data into a JPG file to reduce the amount of disk space needed to store it and re-constructing the image from the compressed JPG data is unlikely to be possible with the very limited computing power of an Arduino.

If you do want to send a photo to an Arduino then you need to tell us approximately how many bytes there are in the image. When we know that we will be better able to advise you.

You also need to tell us what is sending the photo and whether you can make changes to the program that sends it.

...R

13

Robin2:
Images are not broken into 255 parts.

The fundamental unit of data in a PC is a byte which can hold any value from 0 to 255. Larger numbers must be made up from a combination of 2 or more bytes. For example on an Uno or Mega an int (integer) is made from 2 bytes and can hold a value from -32767 to + 32766.

Most colour photos are made up from thousands of pixels with each pixel represented by 3 bytes, one each for the quantity of Red Green and Blue at that point in the image. A black and white image could be represented by a single byte per pixel representing the shade of Grey.

Because photos have thousands of pixels they are usually far too big to fit in the limited memory of an Uno or Mega - for example a very basic 160x120 pixel image will have 19,200 pixels and a minimum of 19200 bytes.

Most cameras compress the image data into a JPG file to reduce the amount of disk space needed to store it and re-constructing the image from the compressed JPG data is unlikely to be possible with the very limited computing power of an Arduino.

If you do want to send a photo to an Arduino then you need to tell us approximately how many bytes there are in the image. When we know that we will be better able to advise you.

You also need to tell us what is sending the photo and whether you can make changes to the program that sends it.

...R

Thanks for lovely explaination!!

Im not quite sure about the size of image right now, but I will get back with it.

A custom built android app is sending that image to arduino and yes I can make changes in the program that sends it.

Further more, I will sending an audio file as well and right now I facing same problems with it as I am facing for image. So would it work for both or I need to adapt different approach for image and audio

14

Count:
So would it work for both or I need to adapt different approach for image and audio

As far as the business of sending data is concerned they are both just files of data.

You can do far more with audio and video on an Android device. I still have no idea why you want to send the data to an Arduino which is puny by comparison.

...R

15

Robin2:
As far as the business of sending data is concerned they are both just files of data.

You can do far more with audio and video on an Android device. I still have no idea why you want to send the data to an Arduino which is puny by comparison.

...R

Minimum size of image will be 200KB and maximum it can go upto 4MB.

THe reason Im doindg this on arduino is becoz I was asked to use Nordic NRF52840 USB dongle for this project but as I got into I realized it was far more difficult than what I thought. Moreover, very less information is available for its programming and you need to buy NRF52840 DK for debugning and there's lot of other hectic things invloved.

So I thought I will use NRF52840 USB dongle, just as serial bluetooth device and let arduino handle rest of every thing.

My question is how do I handle the image data now?

16

Count:
THe reason Im doindg this on arduino is becoz I was asked to use Nordic NRF52840 USB dongle for this project but as I got into I realized it was far more difficult than what I thought. Moreover, very less information is available for its programming and you need to buy NRF52840 DK for debugning and there's lot of other hectic things invloved.

So I thought I will use NRF52840 USB dongle, just as serial bluetooth device and let arduino handle rest of every thing.

The logic of this is all wrong. An Arduino is totally impractical for this. You will just have to figure out how to use the NRF52840 - which I know nothing about myself.

...R

17

Robin2:
The logic of this is all wrong. An Arduino is totally impractical for this. You will just have to figure out how to use the NRF52840 - which I know nothing about myself.

...R

First of all, sorry for very very late reply, Hope you all r doing good and I wish you and your families all the safety.

Exactly that's where the problem is, very low information is available on how to use NRF they don't even have sample examples or building blocks. And that's the reason I put Arduino in to use. Is there any way to get it done?

18

Robin2:
The logic of this is all wrong. An Arduino is totally impractical for this. You will just have to figure out how to use the NRF52840 - which I know nothing about myself.

...R

Do you mean to say that arduino cant receive such large files on serial or it is just impractical because it is unnecessary here?

19

Count:
Do you mean to say that arduino cant receive such large files on serial or it is just impractical because it is unnecessary here?

You can send large files using Arduino if you're smart about it. Using the proper libraries and hardware design, I've helped someone transfer entire JPGs.

Also, for speed, you might consider using a second Arduino to handle the SD card - but only if you need to! I had to implement that design with one of my projects.

20

One thing to remember is that writing to SD can slow down other processing and hence you might miss serial data. The trick would be to slow down the transfer by sending in chunks of e.g. 32 bytes, wait for a reply from the receiver, send the next chunk, wait for reply from receiver and so on.