Serial/Bluetooth data transfer fidelity problem

Hello everyone. I'm hoping you can help me with something I've been fighting with for a week. It's a fairly big project, so I'll try to summarize as best I can and peel out the relevant code for review.

**The issue: **I'm getting all sorts of data corruption in my Android->Arduino data transfers, though the same algorithm works fine for Arduino->Android data transfer.

Hardware: I have a vehicle-mounted Arduino Uno that controls my air bag suspension, interfacing with a bespoke Android app via a HC05 Bluetooth module on the Arduino.

**Intended operation: **Upon a successful connection, the Android app sends to the Arduino a series of 8 unique ID+value data sets in rapid succession, broken up by start/end tokens, and validated by checksums. Each data set is sent twice to allow for one error.

**Actual behavior: **The first two data sets (four id+Value pairs) are successfully read, then the data corruption sets in and no further data passes my error checks as a complete set. Some bytes in the subsequent data are successful, but never a full set of start token, id, value, checksum, and end token. Some corrupted numbers are being read as 8-bit values, but this shouldn't be possible given the way I'm formatting my data.

**Attempted fixes: **I thought this might be a problem with the serial buffer length, so I doubled the buffer size and slowed the Android transfer frequency to once per 100ms. This had no effect.

Additional information: This communication scheme is a revision of a previous (successful) string-parsing communication scheme. The previous scheme was limited to positive numbers only with values less than 128 due to Android/Java's lack of unsigned bytes and Arduino's insistence on them. This new scheme is an attempt to fix that.
I'm currently breaking up signed integers into two 7-bit values to communicate over serial (Java doesn't seem to allow bit masking on the sign bit, so 7 bits is all I get). I use the first of those 7 bits as an indication of byte-pair ordering (1 for first byte, 0 for second byte), and I use the second bit of the first byte as a sign bit. This leaves 11 bits for data.

Relevant Arduino code (full code below):

void receiveData() {
	static bool firstbyte_done = false, id_done = false, value_done = false, checksum_done = false;
	static int id, value, checksum;
	static byte firstbyte;

	byte readbyte;
	int readint;
	
	while (bt_serial.available() > 0 and dataSetReady == false) {
		readbyte = bt_serial.read();
		Serial.print("readbyte ");
		Serial.println(readbyte, BIN);
		
		//first byte checks out
		if ((not firstbyte_done) and (readbyte & 0b01000000)) {
			firstbyte = readbyte & 0b00111111;
			Serial.print("firstbyte ");
			Serial.println(firstbyte, BIN);
			firstbyte_done = true;
			return;
		//second byte checks out
		} else if ((firstbyte_done) and not (readbyte & 0b01000000)) {
			readint = ((int)firstbyte << 6) + (int)(readbyte & 0b00111111);
			Serial.print("READINT ");
			Serial.print(readint, BIN);
			Serial.print(" ");
			Serial.println(readint & 0b0000011111111111); //abs val

			//convert to signed int, using 3rd MSB of first byte as sign bit
			if (readint & 0b0000100000000000) { readint = (readint & 0b0000011111111111) * -1; }
			
			firstbyte_done = false;
		//bad byte received
		} else {
			Serial.println("BAD BYTE PAIR RECEIVED");
			firstbyte_done = false;
			return;
		}

		switch (readint) {
			case START_MARKER:
				id_done = false;
				value_done = false;
				checksum_done = false;
				//Serial.println("START_MARKER");
				break;
			case END_MARKER:				
				if (id_done and value_done and checksum_done 
						and id <= 100 and (id + value == checksum)) {
					receivedData[0] = id;
					receivedData[1] = value;
					dataSetReady = true;
					//Serial.println("END_MARKER");
				} else {
					if (id + value != checksum) { Serial.println("CHECKSUM ERROR"); }
					id_done = false;
					value_done = false;
					checksum_done = false;
				}
				break;
			default:
				if (value_done) { //expecting checksum
					checksum = readint; //check for overflow? not really required if id<=100
					checksum_done = true;
				} else if (id_done) { //expecting value  
					value = (int)readint; 
					value_done = true;
				} else { //expecting id 
					id = (int)readint; //can't be negative since MSB already stripped
					id_done = true;					
				}
				break;			
		}
	} 
}

Full(er) Arduino code: (full code exceeds 9000char forum limit - remainder of code is not relevant for communications)

#include "SoftwareSerial.h"
#include <SoftwareWire.h>
#include <LIS3DH_softwire.h>
#include <math.h>
#include <avr/wdt.h>

//PINS
const int BT_RX = 2;
const int BT_TX = 3;

//COMM CODES
const int comm_FL = 0;
const int comm_FR = 1;
const int comm_RL = 2;
const int comm_RR = 3;
const int comm_tank = 4;
const int comm_pitch = 5;
const int comm_roll = 6;
const int comm_compressor = 7;
const int comm_autolevel = 8;
const int comm_tankmax = 9;
const int comm_tankmin = 10;
const int comm_bagmax = 11;
const int comm_bagmin = 12;
const int comm_deadzone = 13;
const int comm_timer_duration = 14;
const int comm_instantlevel = 15;
const int comm_emergencystop = 16;
const int comm_requestsettings = 99;

//SOFT VARS
int FL_timer_count = 0;
int FR_timer_count = 0;
int RL_timer_count = 0;
int RR_timer_count = 0;
int option_compressor = -1;
int option_autolevel = -1;
int option_tankmax = -1;
int option_tankmin = -1;
int option_bagmax = -1;
int option_bagmin = -1;
int option_deadzone = -1;
int option_emergencystop = -1;
int option_timer_duration = -1;
bool dataSetReady = false;
int receivedData[2];
bool instlevel_in_progress = false;
int level_phase = 0;

//CONSTANTS
const int ATTITUDE_SMOOTHING_FACTOR = 3; //higher factor gives precedence to newest value
const int PRESSURE_SMOOTHING_FACTOR = 3; 
const int ACTIVITY_INTERVAL = 50; 
const int SEND_INTERVAL = 10; 
const unsigned int START_MARKER = 2046; 
const unsigned int END_MARKER = 2047; 
const int ADC_OFFSET = 100;
const float ADC_SCALAR = 3.8;
const int TANK_THRESHOLD = 5;
const int TANK_MAX = 140;
const int BAG_MAX = 70;

class ExponentialFilter {
	private:
		int curr_val;
		int smoothing_factor;
		
	public:
		ExponentialFilter(int smoothing_factor) {
			this->smoothing_factor = smoothing_factor;
			this->curr_val = 0;
		}
		int Current() { return this->curr_val;	}
		void Filter(float filter_val) {
			float smoothing_factor = float(this->smoothing_factor) * 0.01;
			this->curr_val = long(smoothing_factor * filter_val + (1 - smoothing_factor) * float(this->curr_val) + 0.5);
		}
		void SetCurrent(int curr_val) { this->curr_val = curr_val; }
};


SoftwareSerial bt_serial(BT_RX, BT_TX); 
SoftwareWire softwareWire(SOFT_SDA, SOFT_SCL);
LIS3DH_softwire accel = LIS3DH_softwire(&softwareWire);
ExponentialFilter pitch_filter = ExponentialFilter(ATTITUDE_SMOOTHING_FACTOR);
ExponentialFilter roll_filter = ExponentialFilter(ATTITUDE_SMOOTHING_FACTOR);
ExponentialFilter FL_filter = ExponentialFilter(PRESSURE_SMOOTHING_FACTOR);
ExponentialFilter FR_filter = ExponentialFilter(PRESSURE_SMOOTHING_FACTOR);
ExponentialFilter RL_filter = ExponentialFilter(PRESSURE_SMOOTHING_FACTOR);
ExponentialFilter RR_filter = ExponentialFilter(PRESSURE_SMOOTHING_FACTOR);
ExponentialFilter tank_filter = ExponentialFilter(PRESSURE_SMOOTHING_FACTOR);


void setup() {
	wdt_disable(); //watchdog disable
	Serial.begin(19200);
	bt_serial.begin(19200); 
	delay(1000);
	
	if (! accel.begin(0x18)) {   // change this to 0x19 for alternative i2c address
		Serial.println("SETUP: LIS3DH NOT FOUND");
		wdt_enable(WDTO_1S); //enable watchdog to reset arduino
		while (1) {}; //endless loop to trigger watchdog
	}
	accel.setRange(LIS3DH_RANGE_2_G);   // 2, 4, 8 or 16 G
	Serial.println("LIS3DH setup successful");	
	
	send_data(comm_requestsettings, 1);
	send_data(comm_requestsettings, 1);
	send_data(comm_requestsettings, 1);

	pinMode(FL_UP,OUTPUT);
	pinMode(FL_DN,OUTPUT);
	pinMode(FR_UP,OUTPUT);
	pinMode(FR_DN,OUTPUT);
	pinMode(RL_UP,OUTPUT);
	pinMode(RL_DN,OUTPUT);
	pinMode(RR_UP,OUTPUT);
	pinMode(RR_DN,OUTPUT);
	pinMode(COMPRESSOR,OUTPUT);
	
	digitalWrite(FL_UP,HIGH); //HIGH = off
	digitalWrite(FL_DN,HIGH);
	digitalWrite(FR_UP,HIGH); 
	digitalWrite(FR_DN,HIGH);
	digitalWrite(RL_UP,HIGH);
	digitalWrite(RL_DN,HIGH);
	digitalWrite(RR_UP,HIGH);
	digitalWrite(RR_DN,HIGH);
	digitalWrite(COMPRESSOR,LOW); //LOW = off
}


void loop() {
	static long last_activity = 0, last_send = 0;
		
	if (Serial.available())
	bt_serial.write(Serial.read());

	receiveData();
	if (dataSetReady) { 
		Serial.println("RCVD:<"+(String)receivedData[0]+","+(String)receivedData[1]+">");
		enact_instruction(); 
		dataSetReady = false;
		
		if (option_emergencystop == 1) { all_stop(); }
	}
	
	if (millis() - last_activity > ACTIVITY_INTERVAL) {
		last_activity = millis();
		get_pressures();
		get_accel();	
		control_compressor();
		if (instlevel_in_progress or option_autolevel == 1) { instant_level(); }
		
	}
	if (millis() - last_send > SEND_INTERVAL) {
		last_send = millis();
		send_all_data();	
	}
	timercontrol_all_bags();
}

void receiveData() {
	static bool firstbyte_done = false, id_done = false, value_done = false, checksum_done = false;
	static int id, value, checksum;
	static byte firstbyte;

	byte readbyte;
	int readint;
	
	while (bt_serial.available() > 0 and dataSetReady == false) {
		readbyte = bt_serial.read();
		Serial.print("readbyte ");
		Serial.println(readbyte, BIN);
		
		//first byte checks out
		if ((not firstbyte_done) and (readbyte & 0b01000000)) {
			firstbyte = readbyte & 0b00111111;
			Serial.print("firstbyte ");
			Serial.println(firstbyte, BIN);
			firstbyte_done = true;
			return;
		//second byte checks out
		} else if ((firstbyte_done) and not (readbyte & 0b01000000)) {
			readint = ((int)firstbyte << 6) + (int)(readbyte & 0b00111111);
			Serial.print("READINT ");
			Serial.print(readint, BIN);
			Serial.print(" ");
			Serial.println(readint & 0b0000011111111111); //abs val

			//convert to signed int, using 3rd MSB of first byte as sign bit
			if (readint & 0b0000100000000000) { readint = (readint & 0b0000011111111111) * -1; }
			
			firstbyte_done = false;
		//bad byte received
		} else {
			Serial.println("BAD BYTE PAIR RECEIVED");
			firstbyte_done = false;
			return;
		}

		switch (readint) {
			case START_MARKER:
				id_done = false;
				value_done = false;
				checksum_done = false;
				//Serial.println("START_MARKER");
				break;
			case END_MARKER:				
				if (id_done and value_done and checksum_done 
						and id <= 100 and (id + value == checksum)) {
					receivedData[0] = id;
					receivedData[1] = value;
					dataSetReady = true;
					//Serial.println("END_MARKER");
				} else {
					if (id + value != checksum) { Serial.println("CHECKSUM ERROR"); }
					id_done = false;
					value_done = false;
					checksum_done = false;
				}
				break;
			default:
				if (value_done) { //expecting checksum
					checksum = readint; //check for overflow? not really required if id<=100
					checksum_done = true;
				} else if (id_done) { //expecting value  
					value = (int)readint; 
					value_done = true;
				} else { //expecting id 
					id = (int)readint; //can't be negative since MSB already stripped
					id_done = true;					
				}
				break;			
		}
	} 
}

void enact_instruction() {
		
	switch (receivedData[0]) {
		case comm_compressor:
			option_compressor = receivedData[1]; break;
		case comm_autolevel:
			option_autolevel = receivedData[1]; 
			stop_all_bags();
			break;
		case comm_tankmax:
			option_tankmax = receivedData[1]; break;
		case comm_tankmin:
			option_tankmin = receivedData[1]; break;
		case comm_bagmax:
			option_bagmax = receivedData[1]; break;
		case comm_bagmin:
			option_bagmin = receivedData[1]; break;
		case comm_deadzone:
			option_deadzone = receivedData[1]; break;
		case comm_timer_duration:
			option_timer_duration = receivedData[1]; break;
		case comm_FL:
			FL_timer_count = receivedData[1]; break; 
		case comm_FR:
			FR_timer_count = receivedData[1]; break; 
		case comm_RL:
			RL_timer_count = receivedData[1]; break; 
		case comm_RR:
			RR_timer_count = receivedData[1]; break;  
		case comm_emergencystop:
			option_emergencystop = receivedData[1]; break;
		case comm_instantlevel:
			instlevel_in_progress = true; break;
	}
}

Relevant Android code:

void write(Integer sendItem) {

    byte[] sendBytes = splitInt(sendItem);
    Log.d("DEBUG", Integer.toBinaryString(sendBytes[0]) + " " + Integer.toBinaryString(sendBytes[1]));
    //try { mmOutStream.write(sendBytes[0]); mmOutStream.write(sendBytes[1]); }
    try { mmOutStream.write(sendBytes); }
    catch (IOException e) { Log.d("DEBUG","ConnectedThread/write: writing to OutStream", e); }
}

//can only send 11 bits total.
//Java won't let you touch MSB since it's the sign bit, leaving 7 bits per byte.
//2nd MSB is used as position indicator, leaving 6 bits per byte
//3rd MSB of first byte used as new sign bit, leaving 5+6 = 11 bits
private byte[] splitInt(int intToSplit) {
    byte[] splitInt = new byte[2];

    if (intToSplit >= 0) {
        splitInt[0] = (byte) ((intToSplit >>> 6) & 0b00011111 | 0b01000000); //5 bits plus firstbyte bit
    } else {
        intToSplit = -intToSplit;
        splitInt[0]  = (byte)((intToSplit >>> 6) & 0b00011111 | 0b01100000); //5 bits, plus firstbyte bit and sign bit
    }
    splitInt[1] = (byte) (intToSplit & 0b00111111); //6 bits
    return splitInt;
}

Full Arduino code attached, as well as the serial output from my last experiment (note that the tabbed comments were added by me after the fact).

jeep_air.ino (21 KB)

arduino rcvd 1-3-21.txt (16.6 KB)

You have posted a great deal of stuff and I confess I have not read it (sorry for being lazy).

Can you post a few examples of the messages that are being transmitted.

...R

"If I had more time, I would have written a shorter letter." :wink:

Here's a snippet of the Arduino serial output (as posted in the .txt file above), where everything works well. This is near the start of the on-connect data dump.
"readbyte" is the raw data picked up by Serial.read()
"firstbyte" is reported if the first bit indicates it's the most-significant portion of the spit integer
"READINT" is the combination of firstbyte and secondbyte, assuming the position bit checks out on both
Tabbed comments were added by me as I was reviewing the data.

readbyte 1011111
firstbyte 11111
readbyte 111110
READINT 11111111110 2046	start
readbyte 1000000
firstbyte 0
readbyte 1000
READINT 1000 8			id
readbyte 1000000
firstbyte 0
readbyte 0
READINT 0 0			value
readbyte 1000000
firstbyte 0
readbyte 1000
READINT 1000 8			checksum
readbyte 1011111
firstbyte 11111
readbyte 111111
READINT 11111111111 2047	end
RCVD:<8,0>

Here it is starting to fall apart, midway though the connect dump. Note that "readbyte" starts picking up 8 bits, but the output log from the Android confirms that only a maximum of 7 bits is ever sent. Note this isn't the only failure mode - it often reads 7 or less bits, but with incorrect data that fails the first/second byte or checksum checks.

readbyte 111110
BAD BYTE PAIR RECEIVED		missing: start
readbyte 10100000		8 bits???
BAD BYTE PAIR RECEIVED
readbyte 1010
BAD BYTE PAIR RECEIVED
readbyte 11010011		8 bits
firstbyte 10011
readbyte 1000001
BAD BYTE PAIR RECEIVED
readbyte 100100
BAD BYTE PAIR RECEIVED
readbyte 11111110		8 bits

Here's a little more, showing a mix of success and failure:

firstbyte 0
readbyte 1011			
READINT 1011 11			id
readbyte 1000000
firstbyte 0
readbyte 110111
READINT 110111 55		value
readbyte 11010000
firstbyte 10000
readbyte 1011111
BAD BYTE PAIR RECEIVED
readbyte 111111
BAD BYTE PAIR RECEIVED
readbyte 1011111
firstbyte 11111
readbyte 111110
READINT 11111111110 2046	start

Note that the failures are unique to each connect dump, but the general trend is that the first few data sets tend to work and the subsequent sets fail.

mmu:
Here's a snippet of the Arduino serial output

I would prefer to see some examples of the data that is being SENT - just in case the Arduino is getting things mixed up.

...R

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.