Receiving data from android over serial, some data not received

Hello all,

I have a project I am working on where the following is happening:

  1. An android phone is sending strings of data to an arduino over bluetooth, coming into arduino over serial. Arduino baudrate is 9600, bluetooth module (HC-06) also at 9600.
  2. Each loop, the arduino checks if there is serial data available, and reads it char by char
  3. it builds a string char by char, and when it gets to a '/' char, it writes the string to a text file on an SD card.
  4. it clears the string to be built up again by subsequently received data.

I am sending 100 strings total, each with a length of 195 ASCII chars.

All of that is working (mostly) fine. The issue I'm having is that, pretty consistently, on a few lines received I am missing a handful of chars. I seem to always be receiving the delimiter ('/') just fine, but some chars are missing from somewhere in the middle of the group.

When I print out the length of the resulting string after each '/' received, i get all 195's (which is correct) except for maybe 3 or 4 of the 100 lines (that number seems to vary) which print 176. So 19 chars are missing from each of these strings.

I did find one other (hopefully helpful) bit of information out - when I print out Serial.available() with each char received, I am seeing a corresponding spike in how many bytes are available with the lines that are missing chars.

For example, when I print Serial.available() each time the loop runs, I see values that start at 1 and ramp up to 62, maintain at 62 for the majority of the 100 lines sent, then ramp back down to 1 at the end. BUT, around the lines that are missing data, I am seeing that 62 number jump up to 63.

My question is (and it may be stupid), is the serial buffer overflowing when these values read 63, and are therefore not all read before the next set of data comes in? Am I sending data faster than it can be received at my current baudrate?

If so, are there any clever solutions to this problem other than increasing baudrate or decreasing the speed at which I am sending data?

Alternatively, am I fundamentally misunderstanding the problem and need to look somewhere else for a solution?

Thanks in advance. I will post my code if necessary, but I hope the above description does an adequate job on its own.

Best,
Brian

Hi

I had similar issues sending data from an ESP8266 to an Arudino over serial, sometimes it just missed some data, everything worked fine when the data amount was less than the buffers, so it was because of the buffers. On the writing side if the write buffer was full, any new data arriving would push out existing data. At a baud rate of 9600 the write buffer is going to fill very much quicker than it can be emptied.

One option is on the writing side, find out the size of the write buffer than only write up to that amount, then add a delay before continuing with the next batch, this gives time for the data to transfer over and buffers to empty. In my case doing that in small batches of 32 chars with a delay of 5ms between each batch gives me 100% successful transfer, that's at 115200 baud, so you will need to experiment with your delay.

Regards

Phil

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data.

And if that does not solve the problem post your program.

...R

Hi

I was already following the examples there, the problem still happened.

I'm using an Arduino Pro Micro so that might change how some things work, for a start the input buffer is only 32 bytes on Serial1, I found this by seeing what the maximum Serial1.available() would go up to if nothing was read and it stopped at 32, anything after that was lost.

The only way I could get it to work reliably was on the sending device (ESP8266) send blocks of 32 bytes with a delay of a few ms in between, this prevented any buffer overflow.

I think the issue is there is no flow control implemented at the driver level with Arduino. With two wire it can only be software and XON/XOFF bytes anyway, and to support this it has to be written into the Serial.Read/Write code we add in our sketch, which isn't typically done by anyone, i.e. there is no software flow control by default in the Arduino serial libraries, it's up to us.

I suspect typically the buffers are usually read fast enough that things just happen to work by chance than by design, or the data is small enough it fits in the buffers. For more complex sketches, written in a non blocking way on the Serial.read that is moving data that can't all fit into the buffer, that by the time the loop comes back around again to check the buffer and empty it, it's already filled to overflowing.

Regards

Phil

Phil-D:
I was already following the examples there, the problem still happened.

Whose problem are we solving here? I thought it was @btharms' problem.

If you have a problem then start your own Thread and post the code that demonstrates the problem. There is no reason that I am aware of where it would be necessary to send data in chunks of 32 bytes to solve a problem on an Arduino using the code in my examples. Unless, of course, some other part of the Arduino code was preventing my recvWithxxxx() function from being called sufficiently frequently.

I await a response from @btharms

...R

Hi

I was just pointing out that the example code isn't necessarily a fix for the problem if the issue is with buffers and lack of flow control.

There is no reason that I am aware of where it would be necessary to send data in chunks of 32 bytes to solve a problem on an Arduino using the code in my examples

The reason would be if the read buffer is overflowing, as the example code doesn't implement any flow control. A simple example where the only code is reading from the serial port may get away without any flow control and not see any buffer overflows, and it worked fine in my case 9 times out of 10, but it wasn't 100% reliable. I spent hours debugging the problem, I'm just offering some advice and extra info, after all that's what this forum is about, sharing experiences and what worked and what didn't.

Unless, of course, some other part of the Arduino code was preventing my recvWithxxxx() function from being called sufficiently frequently.

Well you just said you could think of no reason why it would be necessary to send data in chunks and then offered one :-). Yes, exactly, and there in lies the problem with simple examples, in the real world if we code for no blocking its for a reason, because we have other things we need done in the loop, and that will of course introduce a slow down to calling the function to receive data. Therefore only the example with no other code in it is guaranteed to work, and that's only by luck rather than by design as it runs fast enough to stop the buffer overflowing. For robust serial data coms some element of flow control is needed. After all that's why there are software and hard flow control options available, the problem is they are not being used in this context.

To be fair your article does say "Note that the 64 byte size of the Arduino serial input buffer does not limit the number of characters that you can receive because the code in the examples can empty the buffer faster than new data arrives." I think it would be good to add something regarding buffer overflows and issues that might arise with real world code doing other things in the loop due to no flow control used.

Regards

Phil

Phil-D:
in the real world if we code for no blocking its for a reason, because we have other things we need done in the loop, and that will of course introduce a slow down to calling the function to receive data.

It would have to slow things down a lot for it to matter. And if you know you need to receive a lot of data then make sure not to slow things down so much.

At 9600 baud it would take about 65 millisecs to fill the 64 byte buffer.

We still have not seen the OP's program.

...R

Thank you guys for the responses. Robin that link is great, apologies for not finding it sooner.
My intuition was kind of aligned with Phil's on this, but even after adding delays on the write side, and trying smaller chunks, I think I've found the culprit on the read side.

To recap, Incoming data looks something like this (simplified but you get the idea)

askjdhhjhflakjhelkrjha/alkjdheeraefwerfawwe/awefagaravfwefeafawg/awgfawefawsfdfvasegg/

I add each character to a string until I see the slash.
I then write the string to a text file on an sd card
i clear the string
keep on keepin on

I think the problem is writing the string to the text file. Maybe it is blocking? When I comment out the line pointed out below I seem to be getting all of the incoming data with nothing missing.

If you see anything else that I'm doing stupidly, I'd appreciate it being pointed out.

Here is a chunk of the pertinent code.

  //create a second serial reader that only does write lines and closes the file when done
  if (Serial.available() > 0 && currentlyWritingToFile) {

    inChar = char(Serial.read());

    switch (inChar) {
      
      case '/': //incoming message is another row of pixel data
        writeToFile(inputString, true); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

        if (stringLen != bytesPerRow) {
          stringError = true; // raise a flag to send an error message back to android
        }

        lineCounter ++; //count one more line for quality check
        rst();
        break;

      case ')': //incoming message is EOM for image text file data
        writeToFile(inputString, false);
        
        currentlyWritingToFile = false;
        
        if (textFile) {
          textFile.close();
        }
        
        if (lineCounter < 99 || stringError == true) { //either a line was too short or there were too few lines
          Serial.print("e"); // send a message to android saying the image didn't come through properly
          stringError = false;
        }
        
        Serial.print(")"); // tell android we are done

        lineCounter = 0;
        rst();
        break;

      case '#': //incoming message is end of initial buffer message
        rst();
        break;

      default:
        inputString += inChar;
        stringLen ++;
        break;
    }
  }
 }
 
 
 void rst() { // clear the string and reset the string len counter
  inputString = "";
  stringLen = 0;
}


void writeToFile(String &s, boolean pLn) {
  if (textFile) {
    
    if(pLn){
      textFile.println(s);
    } else {
      textFile.print(s);
    }
    
    Serial.print("l");
  }

}

btharms:
Here is a chunk of the pertinent code.

Post the complete program. Or, if it is very long, make a special short version that illustrates the problem.

...R

Hey Robin,

Well, I'm stubbornly still hoping the code that I had posted would be sufficient. The whole program is 5-600 lines, and I'm afraid if I take things out to simplify it I may break / fix / or otherwise obscure the issue. I am happy to post the whole thing though if you are willing to look at it.

Is there really not enough information in the code posted? Especially since no data is missing when I comment one line out, and I posted everything that that one line is responsible for.

From the previously posted code, when this line:

writeToFile(inputString, true);

Is commented out, none of the data ends up missing.

That line calls this function, also included in the chunk of code i posted:

void writeToFile(String &s, boolean pLn) {
  if (textFile) {
    
    if(pLn){
      textFile.println(s);
    } else {
      textFile.print(s);
    }
    
    Serial.print("l");
  }

textFile is the currently opened file on the SD card to be written to.

The string argument in this function should always have a length of 195. This is the case when the writeToFile line is commented out. When it is not, occaisionally that string will only have a length of 176 (plus or minus a few characters).

So it seems that the problem is related to writing to the SD card.

For now I'm adding a start character, and every time the terminating character is read, if the string length is wrong it will just ask android for that line again. This should definitely work around my problem, I'm just still not 100% sure what the problem is.

btharms:
So it seems that the problem is related to writing to the SD card.

Yes, probably.

The write time to an SD card is very variable, depending on the card type and datarate I suppose it should do 200bytes in a few milliseconds, but if the card has to do some erase cycle it could be 10's of milliseconds.

If you have the option in the Android software, try adding 100mS delay after it sends a separator '/'

  • assuming that makes it more reliable, it would further indicate that the SD write is taking time and the serial buffer is being overrun.

Yours,
TonyWilk

btharms:
Well, I'm stubbornly still hoping the code that I had posted would be sufficient.

That's fine. I'm not in any hurry.

...R

Hello all,

In case anyone stumbles across this, this was my (high level) solution.

Before, I was sending strings that only had terminating characters that I was looking for on the arduino side. They were sent very rapidly from Android, basically all at once.

I added a start character to the string to be checked on the arduino side as well. This turned out to (probably) not be necessary, but doesn't hurt.

So when I would see the start character, I would start constructing a string from incoming characters over serial. When I hit the terminating character, I would check the length of the string against the expected length of the string. If it was right, I sent a message to android to send the next line. If it was too short, I sent a message telling android to resend the same line. Which is super robust, but actually turns out not so necessary - since android has to wait to receive confirmation that the previous string was received before sending the next string, there were never any issues with the SD card writes taking too long.

I guess the suggestions to add delays on the android side in between sending strings was good advice, I just didn't try a delay that was long enough to solve the problem.

This way is still super robust, and would scale well compared to a hardcoded delay on the android side.

Best,
Brian

btharms:
This way is still super robust, and would scale well compared to a hardcoded delay on the android side.

Thanks for the post; good to hear of your solution.

Yours,
TonyWilk

btharms:
I added a start character to the string to be checked on the arduino side as well.

That's what I suggested in Reply #2

...R