I don't think switching to Sloeber would help in this particular case. The reason is this isn't a problem with the Arduino IDE. This is a problem with the toolchain for compiling and uploading to the ESP32. Arduino IDE and Sloeber both are using the same toolchain. They just pass the appropriate commands to it. So you'll have the same problem no matter which IDE you use so long as they depend on the same toolchain.
For this reason, you would find that the M1 can be used no problem with some other boards.