I2C is okay for a character based LCD but it is waaayyy too slow to drive a TFT, it's by far the worst choice, even with the 400khz high speed version. SPI is a reasonable choice if I/O pins are in short supply, SPI with DMA is even better.
A display shield is the fastest and easiest way to get a TFT LCD up and running on an Arduino. There is an undefined "standard" for this, ultilizing a 20x2 header pin connection to the LCD, a typical example:
The graphics drivers are on the LCD board, common types are the SSD1963 and RA8875. Buydisplay sells both driver types with the standard 20x2 connector and shields as well. It's probably worth a mention that there are far more driver choices in smaller displays but when you get to the higher pixel counts, the field narrows rapidly.
Parallel I/O, 16 bits wide, is usually the fastest way to drive a screen but the Due presents a conundrum in that regard as the designers mapped the physical ports to the digital pins rather poorly, the result being you have to bit twiddle and write to multiple hardware ports to achieve parallel I/O, thereby decreasing the overall efficiency. This is really not a concern if you're using an exsisting library, only if you're writing your own code but I mention this as you said you would be making your own board - and so you could address this deficiency.
Or can I even use the raw display with the Adafruit RA8875 driver ?
You should be able to do that. That said, you can also buy the 800x480 LCD directly from Adafruit and you'd have their support to get things running. While it's certainly more money, it's light years faster delivery and you'd have a real warranty versus a warranty that would require you to return the LCD to China for replacement. Just my 0.02 cents.
The only difference I see see is that the Adafruit implementation supports the SPI interface only. The Buydisplay version of the RA8875 supports both parallel and SPI versions. That's a decision you need to make at some point - but you'd need to test both in order to know which one is faster. Ultimately, I don't know if speed versus I/O count matters to you.