Does your Arduino program work correctly when interfaced with the Serial Monitor rather than your Java code? If not I suggest you write a sketch that does work with the Serial monitor and then get your Java program to interface with that Arduino sketch. That way you have only one area of uncertainty.
The -32 reminds me that I think Java bytes are usually (always?) signed - and they are never signed on the Arduino. So -32 would have the same binary representation as +178. I came across this while using JRuby so I don't know how to deal with the issue in Java. Or there may be a totally different explanation.
Why do you have try/catch surrounding the sleep commands? Is that essential? An error seems unlikely.
What exactly does "serialIn.read(b);" do? (to save me researching the Java documentation - perhaos you have a link).
...R