ESP32 with an Arduino board as the USB interface

Hi, I've got an ESP32 module - Just the module, not on a development board, (but it has its own 600ma voltage regulator) - and I've got it hooked up th an Adafruit Metro M4. I want to use the Metro M4 as a USB to serial port converter so I can re-flash the ESP32 using the Arduino IDE but I've hit some problems.

First off, I can flash the ESP32 if I hook it up to an FTDI USB Serial converter (and hold IO0 low to force it to run the bootloader) - No problem.

Now I have it hooked up to the Metro M4 with the ESP Enable pin, ESP IO0 pin and the VReg enable pin driven by the M4 so the M4 can start the ESP32 and set it into bootloader mode. The code below that runs on the Metro M4 starts the ESP in bootloader mode and then just copies bytes from one serial port to the other. When I open the serial terminal I get the following:

ets Jun 8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2))
waiting for download

This indicates that the ESP32 is in bootloader mode. If I now select an ESP32 board in the Arduino IDE and upload a test sketch (the same one that works fine using the FTDI) I just get this timeout error:

esptool.py v2.3.1
Connecting.....................................................

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header
A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header

Whats also very odd is that if I have an ESP32 board selected in the IDE and I try and open the serial monitor I don't get any data - the serial terminal ONLY displays the startup data when I have the Adafruit Metro M4 board selected in the IDE (It actually works with any NON ESP board I have tried, for example teensy, Uno . .. - The serial data will not appear if any ESP32 board is selected).

I'm confused about what is going on here, firstly why I can't upload a sketch using the Metro M4 as a bridge and secondly why the weird serial port behaviour - I realise they may be connected but I have no clue how to fis this.

I've tried keeping and removing the while(!USB){;} check in the code below and apart from ensuring the serial data is displayed when you open the serial monitorin makes no difference. (And I am making sure the serial monitor is closed when I try and upload the ESP32 sketch)

//USB to Serial bridge for flashing ESP32
#define USB Serial
#define WIRELESS Serial1

void setup() {
  pinMode(PIN_PERIPHERAL_POWER_ENABLE, OUTPUT); //
  pinMode(PIN_ESP32_ENABLE, OUTPUT);
  pinMode(PIN_ESP_BOOT, OUTPUT);
  disablePeripheralPower(); //Keep the ESP32 powered off
  disableESP(); //Set the ESP Enable pin 
  disableESPBootloader(); //Set the ESP boot mode pin
  //Setup the USB and ESP32 Serial ports
  USB.begin(115200);
  WIRELESS.begin(115200);
  enableESPBootloader(); //Set the ESP Bootloader pin to the correct state
  enablePeripheralPower(); //Switch on the voltage regulator
  //delay(5000); //delay, wait for serial port . . .
  //while(!USB){;}
  //Print a startup message
  USB.println("Starting ESP . . .");
  USB.flush();
  enableESP();
}

void loop() {
  // Bridge the two serial ports
  if(USB.available())      WIRELESS.write(USB.read());
  if(WIRELESS.available()) USB.write(WIRELESS.read());
}

void enablePeripheralPower(){  digitalWrite(PIN_PERIPHERAL_POWER_ENABLE,true);}
void disablePeripheralPower(){  digitalWrite(PIN_PERIPHERAL_POWER_ENABLE, false);}
void enableESP(){  digitalWrite(PIN_ESP32_ENABLE, true);}
void disableESP(){  digitalWrite(PIN_ESP32_ENABLE, false);}
void enableESPBootloader(){  digitalWrite(PIN_ESP_BOOT, false);}
void disableESPBootloader(){  digitalWrite(PIN_ESP_BOOT, true);}