That's true but I added the question about the direction 
I forgot to mention before that it works 1 out of 10 times when I power it all on with a 7th (or more) Arduino.
It's exactly this project:
With the exception of:
- it's powered by a Mean Well 5V psu module and a small ams1117 module for the 3,3V.
- a 1000uF electrolytic cap for the ESP on VCC and GND and a small 100nF ceramic in parallel.
The level shifter has indeed 10k pull ups... so I removed the 10k resistors I added previously.
The issue starts by adding a 7th Arduino. Before that it works everytime I power on the circuit.
After the 7th or 8th is attached I have to powercycle it multiple times to get it working.
Sometimes it works on the first try most of the time not.
Thanks for that link!