I'm trying to find out how to make a better webserver from the arduino that would allow many client to connect and have no blocking process.
I would like that the any process that is run by the Arduino does not block the microcontroller more than 20ms.
I've done something similar to what you might need:
https://github.com/stevemarple/WwwServer.
I wanted a data logger to have a built-in web server which would serve saved data files from the SD card. I also wanted the configuration to be stored in a config file on the SD card as for my application I couldn't expect end users to recompile/reflash the Arduino. To avoid interfering with the data logging process I had to split up serving a page into many small tasks which were processed from loop(); this also applies to reading the config file. The maximum time taken for a task is about 50ms. It's not complete but you may find it, or the ideas, helpful. At present it only handles static files, generating dynamic content is planned but I'm working on other things at present.
I don't know what ethernet hardware you are using but the Wiznet5100 supports a maximum of 4 simultaneous connections.