The canvas example does not appear to use PROGMEM except for the fonts? Nor the sort of "transfer the whole image" sort of thing that I thought you were talking about... (Am I looking at the right code? Examples/Adafruit GFX Library/GFXCavnas ? It doesn't look like it does anything like what you describe you're trying to do...)
I am willing to help debug this, from the Pico/Mbed/SPI/whatever point of view, but I have no experience with TFT displays or their use, so my input is going to be largely "theoretical" and require significant input from your side... (or: "Will write code for toys." The last time I did this was quite a while ago ( Arduino to 384 LED display module (MCU Interface) - #4 by westfw ), but it went pretty well...)
What actual display are you using (ILI9341?), and how is it connected? (how does the canvas demo figure this out?) The GFX example will also need the hardware driver for the particular display linked in somehow, right? How is that happening (how does the GFXCanvas example even COMPILE without me specifying a particular display??)
considering the libraries are so terrible for tft screens with refresh rate
Yes, the library is full of special optimization for particular processors, designed to eke out the best performance possible. And there isn't any such optimized code for the rp2040. (and it's entirely possible that having an mBed layer interferes with performance as well.)
But that's a distraction. "Not as fast as I want it to be" is not at all the same thing as "doesn't work at all." If you're going to get anywhere (other than re-writing your code as you've described, which is undesirable and OUGHT to be unnecessary), you need to focus on one problem at a time (and the most serious problem first.)
Yes, it's unfortunate that a product can get released without everything working perfectly. But that's reality, and things will go better for everyone if you aim at helping to make the broken things work instead of just complaining, or even doing your own "work-arounds."
at a meager 63.5mhz it should be well past 60 fps.
(I know I said not to get distracted by performance issues, but... 320x240x24 is ~1.8Mb. At 63.5Mbps, I don't see how you can expect more than about 30fps, and that would need "zero overhead" in the transfer...)