Moving up from UNO and other 328 MCU

I play a lot with UNOs and used Atmel 328 MCUs on my own designed PCBs. Projects are small, not speed or memory intensive.
I've been working with an UNO and a TFT to make a scoreboard with countdown timer and other static text on it. Works well, takes up 56% of program memory.
Now I want to add a SD card, which I've put on a different UNO for testing. Trying to marry the two projects has brought me to that realization that I need more memory and likely more speed.
So: In stock I have a Teensy, Mega, Due, ESP8266 and ESP32, which I bought a while ago, anticipating this day will come. Which one is best likely candidate that will generate the code necessary for the system to work (after I make necessary changes, of course)?
I already tried downloading a naked Teensy and ESP8266 with existing Arduino (C++) code, and got many errors. How easy is it to port from one MCU, the 328, to one of the others?

I have no experience with the Teensy, but have some experience with Mega, and the ESPs. My opinion is that the Mega would be better unless you need the WiFi and/or Bluetooth of the ESPs. And if the Mega's 256K program and 8K SRAM is sufficient.

A note on the SD card modules. Some SD card modules (with level shifters for 5V to 3.3V) will not play well with other devices on the SPI bus. The problem is with the way the level shifter on the SD module is wired. The ones that do not work well have the MISO signal running through the level shifter. That causes the MISO signal to not be released so that the other devices can control it. The modules made by Adafruit (and some others) have the MISO signal (SD card DO) going from the SD card straight to the MISO, bypassing the level shifter. Look at the schematic for the Adafruit module to see what to look for. DO (data out) is the MISO signal.

Thanks, "fun-guy", good info on the SD card... could explain why I didn't get it going with the UNO. I'm using voltage divider for the TFT, not level shifters.
I got my program to compile with the Teensy after changing some pins. The ESP guys are another issue that I would like to work out as I only have one Teensy and a few 8266. Don't need wifi. Might try the Mega, but it's got a big footprint...maybe I can just get the MCU and build a circuit around it? I only need about 10 pins, mostly for the TFT.

The Mega Pro has a much smaller footprint but all of the functionality of a Mega. Smaller even than an Uno. One down side is the 5V of the Mega. SD card and TFT easier to interface with ESP 3.3V (if pins available).

You can use the 5V Mega or its smaller Mega Pro Mini variant, and add 5V to 3.3V buffering with cd74HC4050 powered from 3.3V for SCK, MOSI, and Chip Select, and 1 gate 74AHCT125, with its gate controlled by the CS to the SD card, to bring 3.3V MISO back to 5V, and keep it off the line when the SD is not selected.
Makes for reliable operation, keeps the SD from messing up MISO for other devices, and keeps 5V signals off the SD card's MISO line.

Hey guys. I'm using a Teensy 3.2 with my GLCD display and SD card. Works perfectly. But now to my OTHER project of 16x32 custom matrix display. I made custom MPU/driver boards using an Atmega 328. Works well on sketches that don't do a lot of digital writes, eg my test patterns where I light up sections at a time. For more intensive work, I'm running low on memory and speed is now killing me...I don't have any. Yes, I could tweak the code, use direct register addressing to save cycles but that's a big job. I made my board so that I could swap in a Teensy3.2, but the library I'm using is not compatible. The library is Adafruit's RGBMatrixPanel library. It works with UNO, no work with Teensy. IDEALLY I'd like to use the Teensy but I could make a dongle of sorts with another MCU...but which one?

From the Adafruit GitHub readme.md:

If you need support for RGB888 (24bpp) and need to/can run on Teensy 3.1/3.2/3.5/3.6 or ESP32 chips (not supported by RGB-matrix-Panel), please look at GitHub - marcmerlin/SmartMatrix_GFX: SmartMatrix: Allow using Adafruit::GFX as well as FastLED_NeoMatrix code with SmartMatrix backend, which offers a GFX compatibility layer on top of GitHub - pixelmatix/SmartMatrix: SmartMatrix Library for Teensy 3, Teensy 4, and ESP32

Wow! Lots to digest here. Loaded the libraries, tried a number of example sketches, all give me compile errors, after I do the best I can to configure the sketch for my use. I don't have any "shields" and I don't need this for an ILI9341 TFT which I've already got running with the regular GFX library.

All I need is for RGBMatrixPanel library commands to work with a Teensy3.2 where an Atmega328 worked before. I get the same errors throughout the sketch : D:\ArduinoSketch\libraries\RGBmatrixPanel\RGBmatrixPanel.cpp:141:13: error: cannot convert 'volatile uint8_t* {aka volatile unsigned char*}' to 'volatile PortType* {aka volatile long unsigned int*}' in assignment
latport = portOutputRegister(digitalPinToPort(lat));

I don't see where the SmartMatrix library can help.