I'd say it's more like if you're writing your own game engine, does it matter what language you use for the gameplay logic?
The point behind my question is that the video interfacing itself is likely to be so much work that there isn't much benefit to the Arduino. At that point you may as well use a bare microcontroller. Unless the whole point of this is to produce a TBC specifically for the arduino ecosystem.