Flushing input buffer

Hi guys,
I discovered a very strange behavior of my controlling board. When I send controlling messages (appr. 336-350 X 32byte) to my board, then after a while the execution becomes very slow.
The only way to solve this was, to call "ser.flushInput()" in my control script. flush() or flushOutput() do not help.
My conclusion is, that the input buffer is not getting cleared. Now I am thinking to implement a second security routine in my firmware, which is checking the buffer and if necessary clean it.
This is why I kindly ask, whether "Serial.flush()" the method to solve this problem and whether someone else had done similiar things?!

Part of my python script sending commands to my board:
Now I want to do also a flushInput() in the firmware, because I think it is safer if someone is also using my firmware.

        # concatenate msg and chksum
        output = "%s*%x\r\n" % (str, chk)
        print "Send to control board: " + output

        ser.write(output)
        # flushInput() seems to be extremely important
        # otherwise the APM 2.5 control boards stucks after some commands
        ser.flush()
        ser.flushInput()
        ser.flushOutput()

flush method does not clear the buffer.This behavier was changed prior 1.0 version

flush()
Description

Waits for the transmission of outgoing serial data to complete. (Prior to Arduino 1.0, this instead removed any buffered incoming serial data.)

I don't understand why my buffer is overflowing
(because it would make more sense, that the buffer get freed or overwritten after a command was processed).
Furthermore I cannot find in the Arduino reference a method to get rid of it.

I don't understand why my buffer is overflowing.

Since you didn't post all of your sending code or any of your receiving code, we don't either.

When I send controlling messages (appr. 336-350 X 32byte) to my board

What does this mean? 336 to 350 values that are all 32 bytes in length? That's 11,200 bytes that are going to have a hard time fitting in the 64 byte buffer on the Arduino.

Of course, the Arduino can read serial data far faster than it can arrive, so that shouldn't be a problem. Of course, that requires that you actually do read the serial data fast enough.

Furthermore I cannot find in the Arduino reference a method to get rid of it.

It is a pronoun with no referent. What is "it" that you want to get rid of?

Dumping random amounts of unread serial data hardly seems like the solution to your problem. Whatever it is.

Take a deep breath, and try again.

I don't understand why my buffer is overflowing
(because it would make more sense, that the buffer get freed or overwritten after a command was processed).

How are you processing the incoming data?
Post the code
Serial comunication are slow.To avoid buffer overflow you should read as fast as you can

Made a repo: https://code.google.com/p/rpicopter/source/browse/
And no I am not 350 X sending 32 at once. Every time I send a new 32 byte command.
The only reason I can imagine, is that maybe one byte is not read out and kept in the buffer.

import socket
import json
import serial
from time import *

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('0.0.0.0', 7000))
# whether to use or not to use this timeout is the question?!
#ser = serial.Serial('/dev/ttyACM0', '115200', writeTimeout=0.1)
ser = serial.Serial('/dev/ttyACM0', '115200')

#chksum calculation
def chksum(str):
  c = 0
  for a in str:
    c = ((c + ord(a)) << 1) % 256
  return c

#main loop for receiving a json string
#The checksum will get validated on every important step
def main():
  while True:
    # Wait for UDP packet
    data, addr = sock.recvfrom(128)
    fchksum = int(data[data.find("*")+1 : len(data)])
    data = data[: data.find("*")]

    # Do nothing if checksum of incoming message is wrong
    if chksum(data) != fchksum:
      print "Wrong checksum! Received: %d, calculated: %d" % (fchksum, chksum(data) )
      continue
    else:
      # parse it
      p = json.loads(data)

      # if control packet, send to ardupilot
      if p['type'] == 'rcinput':
        str = "%d,%d,%d,%d" % (p['roll'], -p['pitch'], p['thr'], p['yaw'])
        # calc checksum
        chk = chksum(str)

        # concatenate msg and chksum
        output = "%s*%x\r\n" % (str, chk)
        print "Send to control board: " + output

        ser.write(output)
        # flushInput() seems to be extremely important
        # otherwise the APM 2.5 control boards stucks after some commands
        ser.flush()
        ser.flushInput()
        ser.flushOutput()

main()
#ifndef DEFS_h
#define DEFS_h

// Number of channels
#define APM_IOCHANNEL_COUNT     8

// PID array (6 pids, two for each axis)
#define PID_PITCH_RATE  0
#define PID_ROLL_RATE   1
#define PID_PITCH_STAB  2
#define PID_ROLL_STAB   3
#define PID_YAW_RATE    4
#define PID_YAW_STAB    5

// Motor numbers definitions
#define MOTOR_FL   2    // Front left    
#define MOTOR_FR   0    // Front right
#define MOTOR_BL   1    // back left
#define MOTOR_BR   3    // back right

// Radio min/max values for each stick for my radio (worked out at beginning of article)
#define RC_THR_MIN   1000
#define RC_THR_MAX   1800
#define RC_THR_80P   0.8 * (RC_THR_MAX - RC_THR_MIN) + RC_THR_MIN

#define RC_YAW_MIN   -180
#define RC_YAW_MAX   180

#define RC_PIT_MIN   -45
#define RC_PIT_MAX   45

#define RC_ROL_MIN   -45
#define RC_ROL_MAX   45

uint32_t LSTPKT = 0;
int16_t  CHANNELS[APM_IOCHANNEL_COUNT] = { 0,0,0,0,0,0,0,0 };
char     COMBUF[32];   
int      COMBUFOFFSET = 0;

// PID array (6 pids, two for each axis)
PID pids[6];
// ArduPilot Hardware Abstraction Layer
const AP_HAL::HAL& hal = AP_HAL_AVR_APM2;
// MPU6050 accel/gyro chip
AP_InertialSensor_MPU6000 ins;

#endif /*DEFS_h*/
float wrap_180(float x) {
  return x < -180 ? (x + 360) : (x > 180 ? (x - 360) : x);
}

//checksum verifier
uint8_t verify_chksum(char *str, char *chk) {
  hal.console->printf(str);
  hal.console->printf("\n");

  uint8_t nc = 0;
  for(int i=0; i<strlen(str); i++) 
    nc = (nc + str[i]) << 1;

  long chkl = strtol(chk, NULL, 16); 	// supplied chksum to long
  if(chkl == (long)nc)   				// compare
    return true;

  return false;
}

// Parse incoming text
inline
void ParseInput(int &bytesAvail) {
  for(; bytesAvail > 0; bytesAvail--) {
    char c = (char)hal.console->read(); // read next byte
    if(c == '\n') { // new line reached - process cmd
      COMBUF[COMBUFOFFSET] = '\0'; // null terminator
      // process cmd
      char *str = strtok(COMBUF, "*"); // str = roll, pit, thr, yaw
      char *chk = strtok(NULL, "*"); // chk = chksum

      if(verify_chksum(str, chk)) { // if chksum OK
        char *ch = strtok(str, ","); // first channel
        CHANNELS[0] = (uint16_t)strtol(ch, NULL, 10); // parse       
        for(int i = 1; i < APM_IOCHANNEL_COUNT; i++) { // loop through final 3 channels
          char *ch = strtok(NULL, ",");
          CHANNELS[i] = (uint16_t)strtol(ch, NULL, 10);   
        }
        LSTPKT = hal.scheduler->millis(); // update last valid packet
        memset(COMBUF, 0, sizeof(COMBUF)); // flush buffer after everything
      } 
      else {
        hal.console->printf("Invalid chksum\n");
        memset(COMBUF, 0, sizeof(COMBUF)); // flush buffer after everything
      }
      COMBUFOFFSET = 0;
    }
    else if(c != '\r') {
      COMBUF[COMBUFOFFSET++] = c; 							// store in buffer and continue until newline
    }
  }
}

void setup() {
// TOO LONG FOR FORUM
}

void loop() {
  static float yaw_target = 0;  
  // Wait until new orientation data (normally 5 ms max)
  while (ins.num_samples_available() == 0);

  // serial bytes available?
  int bytesAvailable = hal.console->available();
  ParseInput(bytesAvailable); 	// Parse incoming text

  // turn throttle off if no update for 0.5seconds
  /*if(hal.scheduler->millis() - LSTPKT > 500) {
    CHANNELS[2] = RC_THR_MIN;
  }*/

  long rcthr, rcyaw, rcpit, rcroll; 									// Variables to store radio in 
  // Read RC transmitter 
  rcthr 	= CHANNELS[2]; 
  rcyaw 	= CHANNELS[3];
  rcpit 	= CHANNELS[1];
  rcroll 	= CHANNELS[0];
  // Set an upper limit for the settable throttle (80% of maximum)
  // to allow to counter regulate if copter is changing angle
  /*if(rcthr > RC_THR_80P) {
   rcthr = RC_THR_80P;
   }*/

  // Ask MPU6050 for orientation
  ins.update();
  float roll, pitch, yaw;  
  ins.quaternion.to_euler(&roll, &pitch, &yaw);
  roll 	= ToDeg(roll) ;
  pitch = ToDeg(pitch) ;
  yaw 	= ToDeg(yaw) ;

  // Ask MPU6050 for gyro data
  Vector3f gyro = ins.get_gyro();
  float gyroPitch = ToDeg(gyro.y), gyroRoll = ToDeg(gyro.x), gyroYaw = ToDeg(gyro.z);

  // Do the magic
  if(rcthr > RC_THR_MIN) { 										// Throttle raised, turn on stablisation.
    // Stablise PIDS
    float pitch_stab_output     = constrain_float(pids[PID_PITCH_STAB].get_pid((float)rcpit - pitch, 1), -250, 250); 
    float roll_stab_output 	= constrain_float(pids[PID_ROLL_STAB].get_pid((float)rcroll - roll, 1), -250, 250);
    float yaw_stab_output 	= constrain_float(pids[PID_YAW_STAB].get_pid(wrap_180(yaw_target - yaw), 1), -360, 360);

    // is pilot asking for yaw change - if so feed directly to rate pid (overwriting yaw stab output)
    if(abs(rcyaw ) > 5) {
      yaw_stab_output = rcyaw;
      yaw_target = yaw; 												// remember this yaw for when pilot stops
    }

    // rate PIDS
    long pitch_output 	=  (long) constrain_float(pids[PID_PITCH_RATE].get_pid(pitch_stab_output - gyroPitch, 1), - 500, 500);  
    long roll_output 	=  (long) constrain_float(pids[PID_ROLL_RATE].get_pid(roll_stab_output - gyroRoll, 1), -500, 500);  
    long yaw_output 	=  (long) constrain_float(pids[PID_ROLL_RATE].get_pid(yaw_stab_output - gyroYaw, 1), -500, 500);  

    // mix pid outputs and send to the motors.
    hal.rcout->write(MOTOR_FL, rcthr + roll_output + pitch_output - yaw_output);
    hal.rcout->write(MOTOR_BL, rcthr + roll_output - pitch_output + yaw_output);
    hal.rcout->write(MOTOR_FR, rcthr - roll_output + pitch_output + yaw_output);
    hal.rcout->write(MOTOR_BR, rcthr - roll_output - pitch_output - yaw_output);
  } 
  else {
    // motors off
    hal.rcout->write(MOTOR_FL, RC_THR_MIN);
    hal.rcout->write(MOTOR_BL, RC_THR_MIN);
    hal.rcout->write(MOTOR_FR, RC_THR_MIN);
    hal.rcout->write(MOTOR_BR, RC_THR_MIN);

    // reset yaw target so we maintain this on takeoff
    yaw_target = yaw;

    // reset PID integrals whilst on the ground
    for(int i = 0; i < 6; i++) {
      pids[i].reset_I();
    }
  }
}

All your code. That doesn't compile.

sketch_sep30c.ino: In function 'uint8_t verify_chksum(char*, char*)':
sketch_sep30c:7: error: 'hal' was not declared in this scope
sketch_sep30c.ino: In function 'void ParseInput(int&)':
sketch_sep30c:25: error: 'hal' was not declared in this scope
sketch_sep30c:27: error: 'COMBUF' was not declared in this scope
sketch_sep30c:27: error: 'COMBUFOFFSET' was not declared in this scope
sketch_sep30c:34: error: 'CHANNELS' was not declared in this scope
sketch_sep30c:35: error: 'APM_IOCHANNEL_COUNT' was not declared in this scope
sketch_sep30c:39: error: 'LSTPKT' was not declared in this scope
sketch_sep30c:49: error: 'COMBUF' was not declared in this scope
sketch_sep30c:49: error: 'COMBUFOFFSET' was not declared in this scope
sketch_sep30c.ino: In function 'void setup()':
sketch_sep30c:57: error: 'hal' was not declared in this scope
sketch_sep30c:68: error: 'pids' was not declared in this scope
sketch_sep30c:68: error: 'PID_PITCH_RATE' was not declared in this scope
sketch_sep30c:72: error: 'PID_ROLL_RATE' was not declared in this scope
sketch_sep30c:76: error: 'PID_YAW_RATE' was not declared in this scope
sketch_sep30c:80: error: 'PID_PITCH_STAB' was not declared in this scope
sketch_sep30c:81: error: 'PID_ROLL_STAB' was not declared in this scope
sketch_sep30c:82: error: 'PID_YAW_STAB' was not declared in this scope
sketch_sep30c:86: error: 'GPIO_OUTPUT' was not declared in this scope
sketch_sep30c:91: error: 'ins' was not declared in this scope
sketch_sep30c:91: error: 'AP_InertialSensor' has not been declared
sketch_sep30c:91: error: 'AP_InertialSensor' has not been declared
sketch_sep30c:100: error: 'COMBUF' was not declared in this scope
sketch_sep30c.ino: In function 'void loop()':
sketch_sep30c:106: error: 'ins' was not declared in this scope
sketch_sep30c:109: error: 'hal' was not declared in this scope
sketch_sep30c:119: error: 'CHANNELS' was not declared in this scope
sketch_sep30c:130: error: 'ins' was not declared in this scope
sketch_sep30c:133: error: 'ToDeg' was not declared in this scope
sketch_sep30c:138: error: 'Vector3f' was not declared in this scope
sketch_sep30c:138: error: expected `;' before 'gyro'
sketch_sep30c:139: error: 'gyro' was not declared in this scope
sketch_sep30c:142: error: 'RC_THR_MIN' was not declared in this scope
sketch_sep30c:144: error: 'pids' was not declared in this scope
sketch_sep30c:144: error: 'PID_PITCH_STAB' was not declared in this scope
sketch_sep30c:144: error: 'constrain_float' was not declared in this scope
sketch_sep30c:145: error: 'PID_ROLL_STAB' was not declared in this scope
sketch_sep30c:146: error: 'PID_YAW_STAB' was not declared in this scope
sketch_sep30c:155: error: 'PID_PITCH_RATE' was not declared in this scope
sketch_sep30c:156: error: 'PID_ROLL_RATE' was not declared in this scope
sketch_sep30c:160: error: 'MOTOR_FL' was not declared in this scope
sketch_sep30c:161: error: 'MOTOR_BL' was not declared in this scope
sketch_sep30c:162: error: 'MOTOR_FR' was not declared in this scope
sketch_sep30c:163: error: 'MOTOR_BR' was not declared in this scope
sketch_sep30c:167: error: 'MOTOR_FL' was not declared in this scope
sketch_sep30c:168: error: 'MOTOR_BL' was not declared in this scope
sketch_sep30c:169: error: 'MOTOR_FR' was not declared in this scope
sketch_sep30c:170: error: 'MOTOR_BR' was not declared in this scope
sketch_sep30c:177: error: 'pids' was not declared in this scope

I was not posting the header file. Now it is posted as well. use the code in the repo. It's too long for the forum so I can not post i.e. the setup() function.
And if really someone want to help to find the bug, don't forget to comment out the flushInput(). With this function I don't have this issue. Nevertheless I expect the issue in the firmware.