Random Missing Bytes (arduino mega + processing)

anon77050437 · April 3, 2011, 12:14pm

Board: Arduino Mega (ATmega1280)

Hi Folks,

For a few days now I've been racking my brain, searching the web and the forums and I can't seem to find a solution to a little problem I'm having. The backstory: I'm creating an app in processing that gets data from an XML feed and based on that it will write some data to the arduino for it to then process.

However, I have one small problem - the Arduino side of things seem to be incredibly random in that bytes appear to go missing. I can confirm that the processing side is consistent in what it is sending. I'm sure this has most likely been covered off before but I'd love it if someone could look over my code and help point me in the right direction.

Apologies in advance for the lack of commenting in the code.

Kind regards
Mark.

Processing Code (trimmed down):

String []colour_list = {
  "102,000,053", "054,000,201", "006,000,249", "000,058,255", "000,106,255", "000,154,255", "000,213,255", "000,255,255", 
  "000,255,183", "000,255,111", "000,255,000", "088,255,000", "184,255,000", "259,230,000", "255,187,000", "255,140,000", 
  "255,099,000", "255,047,000", "255,00,000", "195,000,000"
};
int colour_index;

rgbcolour = colour_list[13];

// I've trimmed the rest of the processing code and have pasted in the send
String sendVar = "^" + rgbcolour + "$";
port.write(sendVar);

Processing then sends the following consistently to the Arduino:

^000,255,255$

Arduino Code:

#define START_COLOR_CHAR '^'
#define END_COLOR_CHAR '

The random data after 5 sends from processing to the Arduino:

000,255,255
000,25
02,5
000,255,255
0255

#define COLOR_SIZE 13
#define PIN_RED 9
#define PIN_GREEN 11
#define PIN_BLUE 10

char serialMessage[COLOR_SIZE];
unsigned int readChar;
unsigned int count;
unsigned long color;
unsigned int r;
unsigned int g;
unsigned int b;
boolean readingSerial;

void setup() {
Serial.begin(115200);
readingSerial = false;
}

void loop() {
if (Serial.available() > 0 && !readingSerial) {
if (Serial.read() == START_COLOR_CHAR) {
serialReadColor();
}
}
}

void serialReadColor() {
readingSerial = true;
count = 0;

iniReading:
if (Serial.available() > 0) {
readChar = Serial.read();
if (readChar == END_COLOR_CHAR && count <= COLOR_SIZE) {
//if (readChar == END_COLOR_CHAR) {
goto endReading;
} else {
serialMessage[count++] = readChar;
goto iniReading;
}
}
goto iniReading;

endReading:
readingSerial = false;
serialMessage[count] = '\0';

Serial.println(serialMessage);
//setColor(serialMessage);
}


The random data after 5 sends from processing to the Arduino:

§DISCOURSE_HOISTED_CODE_3§

system · April 3, 2011, 2:02pm

Get rid of the goto statements.

The serialReadColor function should simply read all available serial data, and set a flag to true when the end of the packet is received.

Search the forum for "started && ended" for some code that reads serial data, and appends it to a buffer, after a start-of-packet marker ("^" in your case) is received, until an end-of-packet marker ("S" in your case) is received.

You are only using one flag (started) whereas you really need two (started and ended).

anon77050437 · April 5, 2011, 11:27am

Hey Paul, thanks for the heads up. I found your post on the old forum and tried to implement it and I failed. I think I've been at it for too long and need to come back to it with fresh eyes in a few days. Here's the updated code.

#define START_COLOR_CHAR '^'
#define END_COLOR_CHAR '

and the response written in that last line of code is:

5
000,2,255,25555
000,,255,2555
000255,255
5
000,2,255,255
000,55,255
000,25255,255
5
000,25,255
000,25555,255

000,255,255
000,25,255
0
000,25555,255
000,25,255
0000,255,5,255
000,255255
00,255,,255
000,255,255
00,255,2255
00000,255,255
000,255,55
0000,255,25255
000,255,255
000,,255,2555
000,255,255

#define PIN_RED 9
#define PIN_GREEN 11
#define PIN_BLUE 10

boolean started = false;
boolean ended = false;

char buffer[13];
unsigned char readChar;
unsigned long color;

int serialIn = 0;
unsigned int r;
unsigned int g;
unsigned int b;

void setup() {
Serial.begin(9600);
}

void loop() {

while (Serial.available() > 0) {
readChar = Serial.read();

if (readChar == START_COLOR_CHAR) {
started = true;
ended = false;
}
else if (readChar == END_COLOR_CHAR) {
ended = true;
break;
}
else {
buffer[serialIn] = readChar;
serialIn++;
buffer[serialIn] = '\0';
}
}

if (started && ended){
Serial.println(buffer);
}

}


and the response written in that last line of code is:

§DISCOURSE_HOISTED_CODE_1§

system · April 5, 2011, 11:47am

Is there any delay in the processing code? If not, you may be sending data faster than it can be read, resulting in overflowing the serial receive buffer, resulting in loss of data.

Some things to try. Add:

if(Serial.available() > 0)
{
   Serial.print("Number of characters in buffer: ");
   Serial.println(Serial.available());
}

at the top of loop.

    else {
      if(serialIn <= 12) // Add this!
      {
         buffer[serialIn] = readChar;
         serialIn++;
      }
      buffer[serialIn] = '\0';
    }

Also try increasing the serial baud rate, from 9600 to 57600, without changing how often the data is sent.

anon77050437 · April 5, 2011, 12:04pm

Thanks Paul. I'll give that a try shortly.
As an aside, the frequency of the data send from processing is once every 10 seconds.

system · April 5, 2011, 12:09pm

As an aside, the frequency of the data send from processing is once every 10 seconds.

Then, buffer overflow should not be a problem.

anon77050437 · April 5, 2011, 12:12pm

ok, so I've put in the following (as suggested) and increased baud to 57600:

void loop() {
if(Serial.available() > 0)
{
   Serial.print("Number of characters in buffer: ");
   Serial.println(Serial.available());
}

  while (Serial.available() > 0) {
    readChar = Serial.read();

......

with a few passes I get the following

Nuracters in buffer: 11
Numbcters in buffer: 10
Number of characters in buffer: 1
 characters
 uer of characters in buffer: 14
Numcters in buffer: 15

then I removed the line

   Serial.print("Number of characters in buffer: ");

and got this response after a couple of passes:

Then I put that line back and got the following:

b s in buffer: 14
Numfer: 2
Number of characters in buffer: 11
Number of chbuffer: 15

I'm really not understanding how the outcome can be so different/random every time I run the code.

anon77050437 · April 5, 2011, 12:58pm

now for some more fun and games. I thought maybe the loop is running to fast for everything to be processed. So I threw in a delay at the end of the loop.

delay(5000);

and now for the best part of the response. consistency in the serial.available value but still strange behaviour with the string on the line before that.

Number of charactrsnbfr 5
umber s in buffer: 15
Nbters in buffer: 15
Naracters in buffer: 15
Nmber ofin buffer: 15
Nberrs in buffer: 15
Nber of characters in buffer: 15

I then reinstated the started && ended code

if (started && ended){
    Serial.println(buffer);
  }

with this response

Numbeters in buffer: 15
a000,255,183$

a055,183$

meers in buffer: 15
a000,255,183
a0,255,183
uber of ch
000,255,183
0,Nber of in buffer: 15
00,255,183
025,183
b os in buffer: 15
0,255,183
053
28
Nbcters in buffer: 15
,255,183
,5,183
b  characters in buffer: 15
255,183
,183
umber of characters in buffer: 15
55,183

the complete sketch now looks like

#define START_COLOR_CHAR '^'
#define END_COLOR_CHAR '

#define PIN_RED 9
#define PIN_GREEN 11
#define PIN_BLUE 10

boolean started = false;
boolean ended = false;

char buffer[13];
unsigned char readChar;
unsigned long color;

int serialIn = 0;
unsigned int r;
unsigned int g;
unsigned int b;

void setup() {
Serial.begin(57600);
}

void loop() {
if(Serial.available() > 0)
{
Serial.print("Number of characters in buffer: ");
Serial.println(Serial.available());
}

while (Serial.available() > 0) {
readChar = Serial.read();

if (readChar == START_COLOR_CHAR) {
started = true;
ended = false;
}
else if (readChar == END_COLOR_CHAR) {
ended = true;
break;
}
else {
buffer[serialIn] = readChar;
serialIn++;
buffer[serialIn] = '\0';
}
}

if (started && ended){
Serial.println(buffer);
}
delay(5000);
}

system · April 5, 2011, 1:57pm

The delay() will make things worse, not better. You're now sitting and twiddling your thumbs half the time between sends by the Processing application.

The Serial.available() function will return different values every time it is called. You are calling it three times.

Try this:

void loop()
{
   int sa = Serial.available();
   if(Serial.available() > 0)
   {
      Serial.print("Number of characters in buffer: ");
      Serial.println(sa);
   }

   for(int i=0; i<sa; i++)
   {
    readChar = Serial.read();

    if (readChar == START_COLOR_CHAR)
    {
      started = true;
      ended = false;
    } 
    else if (readChar == END_COLOR_CHAR)
    {
      ended = true;
      break;
    } 
    else
    {
      buffer[serialIn] = readChar;
      serialIn++;
      buffer[serialIn] = '\0';
    }
  }

  if (started && ended)
  {
    Serial.print("buffer contains: [");
    Serial.print(buffer);
    Serial.println("]");
  }
}

anon77050437 · April 6, 2011, 10:22am

Thanks Paul, I've tried your suggested code and get the following on the first send of data from processing.

uber of characters in buffer: 1
Number of characters in buffer: 14
buffer contains: [,183]
buffer contains: [,183]
buffer cont
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,183]
bus: [,183]
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,
fer contains: [,183]
ue otis ,8]bfe oan:,8]bfrcn]
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,183]
buffer contains: [,183]
buffer containuffer conta

this repeats until the second send from processing

Number of characters in buffer: 2
buffer contains: [,183]
Number of characters in buffer:contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [18]bfe[183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [13
ufrotis[8]
fe otis 8]bffer containbfer cont
buffer contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [1833]
buffer contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer  contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [183]
buffer contains: [183]
buf: [183]

processing is sending

^000,255,183$

full sketch

#define START_COLOR_CHAR '^'
#define END_COLOR_CHAR '

#define PIN_RED 9
#define PIN_GREEN 11
#define PIN_BLUE 10

boolean started = false;
boolean ended = false;

char buffer[13];
unsigned char readChar;
unsigned long color;

int serialIn = 0;
unsigned int r;
unsigned int g;
unsigned int b;

void setup() {
Serial.begin(57600);
}

void loop()
{
int sa = Serial.available();
if(Serial.available() > 0)
{
Serial.print("Number of characters in buffer: ");
Serial.println(sa);
}

for(int i=0; i<sa; i++)
{
readChar = Serial.read();

if (readChar == START_COLOR_CHAR)
{
started = true;
ended = false;
}
else if (readChar == END_COLOR_CHAR)
{
ended = true;
break;
}
else
{
buffer[serialIn] = readChar;
serialIn++;
buffer[serialIn] = '\0';
}
}

if (started && ended)
{
Serial.print("buffer contains: [");
Serial.print(buffer);
Serial.println("]");
}
}

system · April 6, 2011, 11:47am

Once the end of packet marker is received, and the data used (or just printed in this case), the buffer and flags and index need to be reset.

  if (started && ended)
  {
    Serial.print("buffer contains: [");
    Serial.print(buffer);
    Serial.println("]");

    // Add these
    started = false;
    ended = false;
    buffer[0] = '\0';
    serialIn = 0;
  }

system · September 17, 2011, 9:19pm

Hello there!

Actually I am facing the exactly same issue. I am using Arduino mega 1280 and sending data from PC to arduino via USB cable.
The only difference is that I am using JSon format instead of XML.

Arduino receives random characters, some are correct but most are not. Seems that it only receives the characters partially as well.

My string has about 150 chars and begins and ends with "{" and "}" respectively.
I have set the serial at 9600 in both sides (PC-Arduino).

My code is pretty much the same as yours.

Do you know if there's anything else that has to be done in order to sync the data? Or should I just resend it until it works?

Thank you gauys!

system · September 17, 2011, 9:29pm

My code is pretty much the same as yours.

Pretty much != exactly the same. So, post YOUR code.

My string has about 150 chars and begins and ends with "{" and "}" respectively.

The serial receive buffer on the Arduino can only hold 128 characters. If more than that are sent, some may be discarded.

Seems that it only receives the characters partially as well.

What, like half an a is received? This makes no sense.

I have set the serial at 9600 in both sides (PC-Arduino).

Some reason from dragging along at a snails pace? Pick up the speed.

system · September 19, 2011, 1:29am

Actually you have just solved my issue.

When I raised the speed to 57600 and limited my string to 128 chars the problem got solved.

Thanks a lot!

retrolefty · September 19, 2011, 3:42am

What, like half an a is received? This makes no sense.

Your in top form tonight Paul.

Topic		Replies	Views
Why the Processing datas are not read well in the Arduino monitor? NOT RESOLVED Interfacing w/ Software on the Computer	8	2584	May 6, 2021
Sending byte array over Serial comm Programming	31	1160	October 28, 2024
Arduino stops receiving serial characters into the buffer after some are received (less than 5 in the buffer) Programming	38	1398	June 23, 2023
Serial communication speed issue? how to delay buffer Programming	23	678	March 2, 2024
Problem parsing serial data Programming	24	1157	February 26, 2024

Random Missing Bytes (arduino mega + processing)

Related topics