Just noticed that there is a power down pin (32), so I tried the following:
// GPIO 32 is NOT a RTC pin
digitalWrite(power_down_pin, HIGH);
gpio_hold_en(GPIO_NUM_32);
esp_light_sleep_start(); // enter light sleep
Serial.println("Returning from light sleep");
gpio_hold_dis(GPIO_NUM_32);
digitalWrite(power_down_pin, LOW);
I now get 6mA and a very fast wakeup.