You don't need to add delays. You need to read bytes, using Serial.available() to check if any is there, right now, and keep reading until you have all you need. That's what you need to do. If you fudge it with a delay, one day the delay will be wrong and you will get rubbish data.