To me it suggests there is nothing wrong with the arduino, but rather that your C# program is not set to run at the correct baud rate - in which case there arduino recieves a series of nonsense bytes instead on 1. It processes the first and decides that it is no <128, so processes the next, and so on until presumably it gets to the fifth at which point that one happens to be <128, so it sends it back. That would account for the 5s delay.
But as you haven't said what it sends back, or what you send, it is impossible to know that for sure.
Does it work at the higher BAUD rates using the Arduino Serial monitor? (My bet is that it does). If so, then it is nothing to do with the Arduino.