I figured out my issue, why I needed the delays (or didn't need them), and got 115200 working.
"if (!Serial.available()) {}" should have been "while (!Serial.available()) {}", then the delays are unnecessary and everything works full speed!
Now does anyone know the packet format of bob-light? It is still not working right, but my self-made app works, it just has a while to go before it can average pixels as well as boblight already does.