I'm not sure R4 thru R7 are needed. They don't seem to do anything (are they part of the switches?).
I wouldn't bother connecting USB UART chip's RTS and CTS to ESP. There's no need. You need to connect RTS to other pins.
Regarding "direction and pulldown", what are the caps doing there? Every time you toggle the IO pin, it is going to cause temporary short circuit. You need a resistor in-line with the cap.
The eagle you use has a strange shade of green. I'm still using the pre autodesk version. I might decide to try the new version. You have to get an autodesk account, I suppose. Have you used older eagle versions?