You are connecting every time round the loop but never ending the connection, you need to bear in mind that there is a very limited number of TCP connections and once exceeded the client will need to make a decision on which one to replace with a new connection. It would be better if the first client.connect was not there - it should be connected in setup and only reconnect if it is not connected.
What other users of the WIFI network are there, are there any repeaters, if on 2.4GHz are there any cordless phones, microwaves or anything that could put noise on the frequency in use causing delays?