Should i remove all the delays?
Yes
If you need portions of your program to execute for periods of time but don't want pauses to prevent other functions, reading of sensors etc affected then look at using the principle in the BlinkWithoutDelay example in the IDE.
Note the start time of an activity and periodically check whether the required time has elapsed. If not, do something else and come back later to check again, otherwise react to the time elapsing.