Reliable Data Transfer with Arduino and HC-06 module

Hi All.
First time asking on the forum, but I have been using many of the pages on here for help.
I am trying to communicate from a computer to an arduino nano over bluetooth for an LEDCube project I have been working on and off for over a year. I have managed to get data sent from the computer to the arduino using an HC-06 module by DSD tech, but I am having reliability issues, resulting in limited effective bandwidth. Data is coming through, it is just really slow, and bluetooth at this speed breaking up this badly does not seem right.

I was using SoftwareSerial, but due to the slowdown, decided to try a mockup board running on the rx/tx pins instead. There is no difference in reliability. Sometimes it works on the 2nd attempt, others it is 30 or 40 tries. Any help would be greatly appreciated. As to one of the questions asked in response to a similar question, no, I do not want to just use a wire. I plan on having this as a display thing, and the power wire is enough, and I don't want to run a wire across my room to connect to it.

Extremely pared down arduino code for bluetooth functionality:

#define baudRate 115200

//vars for BT
char c = ' ';
boolean NL = true;
char BTBuff[32]; //max 32 chars. 2 reserved for carraige return and new line
char BTInput[32];
short BTBuffIndex=0;

void setup() {
  // put your setup code here, to run once:
  //serial for computer and BT
  Serial.begin(baudRate); //Start serial comms
  Serial.println("started");
}

void loop() {
  //handle bluetooth comms
  updateBT();

}


void processBT(){
  const char *ptr =strchr(BTBuff,'*');

  char *command;
  bool checkSumStatus = false;
  if(ptr){
    int stringSize = ptr-BTBuff;
    //get checksum chunks and check
    char *str = strtok(BTBuff,"*");
    char *checkSumStr = strtok(NULL,"*");  
  
    unsigned int sum = atoi(checkSumStr);
    
    unsigned int calcCheckSum = Fletcher16(str,stringSize);

    if(sum == calcCheckSum){
      Serial.println(sum);
      checkSumStatus=true;
    }
    else{
      Serial.println("5");
      checkSumStatus=false;
    }
  }  
}

void updateBT(){
  // Read from the Bluetooth module and send to the Arduino Serial Monitor
  
  if (Serial.available()>0)
  {  
      c = Serial.read(); //get character
      BTBuff[BTBuffIndex]=c; //add to buffer
      
      if(BTBuffIndex>0){ 
        //check last 2 characters, if return, newline then send substring of useable characters through serial. reset index
        
        if(BTBuff[BTBuffIndex-1] == '\r'){
          if(BTBuff[BTBuffIndex]=='\n'){
            processBT();
            BTBuffIndex = -1;// set negative so when added in next line, is at 0;
          }
        }
      }

      //every character, increase the active index
      BTBuffIndex++;

      //if active index too big for content, wrap back to beginning (prevent overwriting memory)
      if(BTBuffIndex>=32){
        BTBuffIndex=0;
      }
  }
}

int Fletcher16(char *str,int stringSize){
  unsigned int sum1 = 0;
  unsigned int sum2 = 0;
  for(int i =0;i<stringSize;i++){
    sum1 = (sum1 + str[i]) % 255;
    sum2 = (sum2 + sum1) % 255;
  }  
  //Serial.print("checksum: ");
  //Serial.print(sum1);
  //Serial.print(" ");
  //Serial.println(sum2);
  return (sum1 << 8) | sum2;
}

BluetoothConnector project in JAVA (what I am controlling the cube from). Uses BlueCove maven library

package bluetooth;

//https://stackoverflow.com/questions/33473926/best-practice-java-serial-bluetooth-connection-hc-05

import java.awt.FlowLayout;
import java.awt.Font;
import java.awt.GridLayout;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Scanner;
import java.util.logging.Level;
import java.util.logging.Logger;

import javax.bluetooth.DeviceClass;
import javax.bluetooth.DiscoveryAgent;
import javax.bluetooth.DiscoveryListener;
import javax.bluetooth.LocalDevice;
import javax.bluetooth.RemoteDevice;
import javax.bluetooth.ServiceRecord;
import javax.bluetooth.UUID;
import javax.microedition.io.Connector;
import javax.microedition.io.StreamConnection;
import javax.swing.DefaultListModel;
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JList;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
import javax.swing.JTextField;
import javax.swing.ListSelectionModel;

public class BluetoothConnector {
	private static boolean scanFinished = false;
	private static boolean connected = false;
	private static ArrayList<RemoteDevice> matchedDevices;
	private static ArrayList<String> URLs;
	private static StreamConnection streamConnection;
    private static OutputStream os;
    private static InputStreamReader is;
    
    private static JTextArea historyArea= new JTextArea();;
    
    private static String serialMonitorString = "";
    private static Scanner s;
	
    //connect calls autoconnect if it can, otherwise starts manual
    public static void connect(String filter) {
		if(!trySavedConnection()) {
			setupWindow(filter);
		}
	}
	
    //closes all connections
	public static void disconnect() throws IOException {
		
		try {
			//close streams, scanner, and shows disconnected
			s.close();
			
			os.close();
	        is.close();
	        streamConnection.close();

		}catch(NullPointerException e) {
		}
		        
        
        connected = false;
	}
	
	//returns connected variable
	public static boolean connected() {
		return connected;
	}

	//write string to bluetooth
    public static void write(String text) throws IOException {
		if(!connected) {
			return;
		}
		
		//uses carriage return and newline to end strings
		String preppedText = text+"\r\n"; 
		
		updateHistoryAreaText("  > "+preppedText);
		
		os.write(preppedText.getBytes()); 
	}
    
    public static int reliableWrite(String text) throws IOException{
    	if(!connected) {
    		return -1;
    	}
    	
    	
    	
    	//uses carriage return and newline to end strings
    	int checksum = Fletcher16(text);
		String preppedText = text+"*"+checksum+"\r\n"; 
	
		updateHistoryAreaText("  > "+preppedText);
			
    	int fails = -1;
		
    	String result = "";
    	while(!result.equals(Integer.toString(checksum))) {
    		os.write(preppedText.getBytes());
        	
        	result = read();
        	if(result.equals("")) {
        		try {
					Thread.sleep(10);
				} catch (InterruptedException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
        	}
        	
    	}
    	
    	return fails;
    }
	
    //read string from bluetooth
	public static String read() throws IOException {
		String result = "";
		
		if(!connected) {
			return "";
		}
			
		//if result has content, get it.
		char[] charBuffer = new char[255];
		
		int charsRead=-1;
		
		if(is.ready()) {
			charsRead=is.read(charBuffer);
			
		}
		
		if(charsRead!=-1) {
			for(int i=0;i<charBuffer.length;i++) {
				if(i>1) {
					if(charBuffer[i]=='\n' && charBuffer[i-1]=='\r') {
						break;
					}
				}
				result +=charBuffer[i];
				
			}
			
		}
		
		
		if(!result.equals("")) {
			updateHistoryAreaText(result+"\n");
		}
		
		result=result.strip();
		return result;
		
	}
	
	//launch serial monitor
    public static void serialMonitor() {
    	
    	if(!connected) {
    		System.out.println("Not connected");
    		return;
    	}
    	JDialog serialMonitor = new JDialog();
		serialMonitor.setTitle("SerialMonitor");
		//serialMonitor.setModal(true);
		serialMonitor.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
		serialMonitor.setSize(750, 400);
		
		//history area shows serial comms
		historyArea = new JTextArea();
		historyArea.setText(serialMonitorString);
		historyArea.setSize(600,300);
		
		historyArea.setLineWrap(true);
		historyArea.setEditable(false);
		historyArea.setVisible(true);
		historyArea.setFont(new Font(Font.MONOSPACED,Font.PLAIN,12));
		
		//put historyArea in scroll
		JScrollPane historyScroll = new JScrollPane(historyArea);
		historyScroll.setVerticalScrollBarPolicy(JScrollPane.VERTICAL_SCROLLBAR_ALWAYS);
		historyScroll.setHorizontalScrollBarPolicy(JScrollPane.HORIZONTAL_SCROLLBAR_NEVER);
		
		JTextField output = new JTextField();
		output.addActionListener(new ActionListener() {

			//actionListener listens for enter by default. Try to write and wipe the text
			@Override
			public void actionPerformed(ActionEvent e) {
				String text = output.getText();
				
				output.setText("");
				
				try {
					write(text);
				} catch (IOException e1) {
					e1.printStackTrace();
				}
				
			}
			
		});
		
		GridLayout layout = new GridLayout(0,1);
		
		serialMonitor.setLayout(layout);
		
		serialMonitor.add(historyScroll);
		serialMonitor.add(output);
		
		serialMonitor.pack();
		serialMonitor.setVisible(true);
    	
    	//make listener and have both sender and listener always save to here somehow
    }
	
    //attempts to connect to url.txt
	private static boolean trySavedConnection() {
		Scanner URLGetter;
		try {
			//get saves url
			File file = new File("url.txt");
			URLGetter = new Scanner(file);
			
			String URL = "";
			while(URLGetter.hasNextLine()){
				URL = URLGetter.nextLine();
			}
			
			URLGetter.close();
			
			//connect using url if not empty
			if(!URL.equals("")) {
				tryConnect(URL);
				if(connected) {
					return true;
				}
				else {
					return false;
				}
			}
			else {
				System.out.println("No saved connection");
				return false;
			}
			
			
			
			
		} catch (FileNotFoundException e) {
			System.out.println("url.txt does not exist");
			return false;
		}
	}
	
	//launches full setup window and establishes connection
	private static void setupWindow(String filter) {
		JDialog setupDialog = new JDialog();
		setupDialog.setTitle("Bluetooth Selector");
		//forces program to stall when this is launched
		setupDialog.setModal(true);
		setupDialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
		
		DefaultListModel<String> scanListModel = new DefaultListModel<String>();
		DefaultListModel<String> scanServiceListModel = new DefaultListModel<String>();
		
		JList<String> scanList = new JList<String>(scanListModel);
		scanList.setSelectionMode(ListSelectionModel.SINGLE_SELECTION);
		scanList.setLayoutOrientation(JList.VERTICAL);
		
		scanList.addMouseListener(new MouseAdapter() {
		    public void mouseClicked(MouseEvent evt) {
		    	
		        @SuppressWarnings("unchecked")
				JList<String> clickedList = (JList<String>)evt.getSource();
		        //get element clicked on, and run service scan on it.
		        if (evt.getClickCount() == 2) {
		            // Double-click detected
		            int index = clickedList.locationToIndex(evt.getPoint());
		            try {
						serviceScan(matchedDevices.get(index));
						scanServiceListModel.removeAllElements();
						
						//service scan populates URLs
						if(URLs.size()>0) {
							for(int i=0;i<URLs.size();i++) {						
								scanServiceListModel.addElement(URLs.get(i));
							}
						}
						else {
							System.out.println("no Services available, check for unclosed connections or restart");
						}
						
						
					} catch (Exception ex) {
						// TODO Auto-generated catch block
						Logger.getLogger(BluetoothConnector.class.getName()).log(Level.SEVERE,null,ex);
					}
		        }
		    }
		});
		
		scanList.setVisibleRowCount(5);
        JScrollPane scanListScrollPane = new JScrollPane(scanList);
        
        //button to initialize scan for RGBCube
		JButton scanButton = new JButton("Scan");
		scanButton.addActionListener(new ActionListener() {

			@Override
			public void actionPerformed(ActionEvent e) {
				try{
					scan(filter);
					//scan populates matchedDevices. Add to list all devices
					scanListModel.removeAllElements();
					for(int i=0;i<matchedDevices.size();i++) {						
						scanListModel.addElement(matchedDevices.get(i).getFriendlyName(false) + ":" + matchedDevices.get(i).getBluetoothAddress());
					}
					
				}catch(Exception ex) {
					Logger.getLogger(BluetoothConnector.class.getName()).log(Level.SEVERE,null,ex);
				}
			}
		});
		
		
		
		JList<String> scanServiceList = new JList<String>(scanServiceListModel);
		scanServiceList.setSelectionMode(ListSelectionModel.SINGLE_SELECTION);
		scanServiceList.setLayoutOrientation(JList.VERTICAL);
		
		scanServiceList.addMouseListener(new MouseAdapter() {
		    public void mouseClicked(MouseEvent evt) {
		    	
		    	//get service clicked on
		        @SuppressWarnings("unchecked")
				JList<String> clickedServiceList = (JList<String>)evt.getSource();
		        if (evt.getClickCount() == 2) {
		            // Double-click detected
		            int index = clickedServiceList.locationToIndex(evt.getPoint());
		            try {
		            	
		            	String URL = URLs.get(index);
		            	//try to connect to url clicked on
		            	tryConnect(URL); 
						if(connected) {
							//if connected, wipe serial monitor string, save url to file, close dialog
							serialMonitorString="";
							System.out.println("Connected");
							
							File oldFile = new File("url.txt");
							oldFile.delete();
							
							File file = new File("url.txt");
							
							FileWriter f = new FileWriter(file,false);
							f.write(URL);
							f.close();
							
							setupDialog.dispose();
							
						}
						else {
						}
					} catch (Exception ex) {
						// TODO Auto-generated catch block
						Logger.getLogger(BluetoothConnector.class.getName()).log(Level.SEVERE,null,ex);
					}
		        }
		    }
		});
		
		scanServiceList.setVisibleRowCount(5);
        JScrollPane scanServiceListScrollPane = new JScrollPane(scanServiceList);
		

		setupDialog.setLayout(new FlowLayout());
		setupDialog.add(scanButton);
		setupDialog.add(scanListScrollPane);
		
		setupDialog.add(scanServiceListScrollPane);
		
		
		setupDialog.pack();
		setupDialog.setVisible(true);
	}
    
	//scan for bluetooth devices
	private static void scan(String nameFilter) throws Exception{
		scanFinished = false;
		
		matchedDevices = new ArrayList<RemoteDevice>(); 
		
		LocalDevice.getLocalDevice().getDiscoveryAgent().startInquiry(DiscoveryAgent.GIAC, new DiscoveryListener() {

			@Override
			public void deviceDiscovered(RemoteDevice btDevice, DeviceClass cod) {
				try {
					//if device found, get name, and add to list of devices
					String name = btDevice.getFriendlyName(false);
					System.out.format("Pre filter: %s (%s)\n", name, btDevice.getBluetoothAddress());
					if(nameFilter.equals("")) {
						matchedDevices.add(btDevice);
					}
					else if(name.matches(nameFilter)) {
						matchedDevices.add(btDevice);
					}
				}catch(IOException e) {
					e.printStackTrace();
				}
			}

			@Override
			public void servicesDiscovered(int transID, ServiceRecord[] servRecord) {}

			@Override
			public void serviceSearchCompleted(int transID, int respCode) {}

			@Override
			public void inquiryCompleted(int discType) {
				scanFinished = true;
			}
		});
		
		while(!scanFinished) {
			Thread.sleep(500);
		}
		
	}
	
	//scan for bluetooth services
	private static void serviceScan(RemoteDevice hc06Device) throws Exception {
		
		URLs = new ArrayList<String>();
		
		UUID uuid = new UUID(0x1101);
		UUID[] searchUUIDSet = new UUID[] {uuid};
		int[] attrIDs = new int[] {
				0x0100
		};
		scanFinished = false;
		//scan for services
		LocalDevice.getLocalDevice().getDiscoveryAgent().searchServices(attrIDs, searchUUIDSet,
				 hc06Device, new DiscoveryListener() {
	                    @Override
	                    public void deviceDiscovered(RemoteDevice btDevice, DeviceClass cod) {
	                    }

	                    @Override
	                    public void inquiryCompleted(int discType) {
	                    }

	                    @Override
	                    public void serviceSearchCompleted(int transID, int respCode) {
	                        scanFinished = true;
	                    }

	                    @Override
	                    public void servicesDiscovered(int transID, ServiceRecord[] servRecord) {
	                    	//if service found, get url and save it
	                    	System.out.println("Scanning record ");
	                    	for (int i = 0; i < servRecord.length; i++) {
	                        	
	                        	String URL = servRecord[i].getConnectionURL(ServiceRecord.NOAUTHENTICATE_NOENCRYPT, false);
	                        	if (URL != null) {
	                            	URLs.add(URL);
	                            	break; //take the first one
	                            }
	                        }
	                    }
	                });
		 while (!scanFinished) {
	            Thread.sleep(500);
        }
		 
	}
	
	//attempts connection to url
	private static void tryConnect(String URL) {
        System.out.println("URL "+URL);
        
        //address btspp://0014030559FA:1;authenticate=false;encrypt=false;master=false
        //if you know your hc05Url this is all you need:
        
    	
        try {
        	//open connection, setup IO lines
        	streamConnection = (StreamConnection) Connector.open(URL);
            os = streamConnection.openOutputStream();
            InputStream stream = streamConnection.openInputStream();
            is = new InputStreamReader(stream);
			//is = streamConnection.openInputStream();
			connected = true;
			
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
	
	//updates text area of serial monitor
	private static void updateHistoryAreaText(String string) {
		if(string.equals("")) {
			serialMonitorString = "";
		}
		else{
			serialMonitorString += string;
		}
		historyArea.setText(serialMonitorString);
		historyArea.setCaretPosition(historyArea.getDocument().getLength());
		
	}

	//from https://en.wikipedia.org/wiki/Fletcher%27s_checksum
	private  static int Fletcher16(String message) {
		int sum1 = 0;
		int sum2 = 0;
		
		for(int i=0;i<message.length();i++) {
			sum1 = (sum1 + message.charAt(i)) %255;
			sum2 = (sum2+sum1) %255;
		}
		
		//System.out.println("Checksum = " +sum1+ " "+sum2);
		//System.out.println("shift combine = " + ((sum1<<8)|sum2));
		return (sum1<<8)|sum2;
	}

}

Main Java class to send data to arduino

package bluetooth;

import java.io.IOException;

public class Main {

	public static void main(String[] args) throws IOException {
		
		

		BluetoothConnector.connect(""); //blocking
		
		System.out.println("hello world");
		
		BluetoothConnector.serialMonitor(); //nonblocking, shows serial monitor, send and receive
		
		
		if(BluetoothConnector.connected()) {
			for(int i = 0;i<64;i++) {
				int fails = BluetoothConnector.reliableWrite("i:"+i);
				if(fails!=-1) {
					System.out.println("fails: "+fails);
				}
				
			}			
		}

		BluetoothConnector.disconnect(); //must disconnect on close to prevent hanging
		
		
	}

}

Thank you for making it this far. Please let me know if you need any more info.

For reliability, you need a communications protocol that can at least detect errors and respond with a request for retransmission.
Paul

I incorporated Fletcher16 into it to create the reliable data transfer. I have reliableWrite in the Java program which add * followed by the fletcher16 checksum. In Arduino, this is then split off, and the checksum is calculated again. If the 2 match, the checksum is sent back as a unique ok, otherwise it sends "5" just to make something go back. If the returned checksum does not match, Java writes again.

How many fails do you get for one message? How many retries?
Paul

I get between 1 and 40 retries before it goes through. I do not let it flat out fail. It will resend until it works

Yes, that is terrible. Back in the 70's with modems and leased telephones lines, one retransmission out of a thousand was rare. I think it was 5 retransmissions caused the connection to be dropped and an error was posted requiring the connection to be examined.
Paul

Not sure if this helps, but I ran 2 tests to see how bad the error is. It is much worse than I said before, but it also has a weird shape. image

I have no clue why it would do this ramping thing, I cannot find any info about it in Java, and I did not make it do that. It somehow recovers

It also stays low after that peak. I made it do another 64 after the first without closing the connection, and the highest it went was 6