it seems through the serial port there are a bunch of random data slowing my connection down
Make sure that the phone app is sending the commands to the BT with out line endings.
You should be able to echo the Serial.read() values back to the phone with Serial.write() commands and see what the "random data" is. Perhaps if you shared the findings with us, we could help.