Using the Mege 2560, I would like to run 12 16X16 WS2812b LED panels, total or 3072 LEDs. Whether the Mega is attached to the IDE or not, #define NUM_LEDS 2696 is the highest number I can enter before getting a memory error.
Is this an IDE 'cap' or is it really the limit of the Mega?
Would an ESP32 handle it?
Since you did not include any code I will assume that you are using the FastLED library. With that library you create an array of CRGB for the LEDs. Each element of the array represents 1 pixel so has 3 bytes (R, G, B). The Mega has 8192 bytes of SRAM (where the array is stored). So the most LEDs is 2730, but the rest of the sketch will take some SRAM so the real maximum will be less. That is why your max is 2696. 3072 pixels would take 9216 bytes + enough for the programs use.
How much SRAM does an ESP32 have? Will a 3.3V ESP32 work with WS2812 strips?