I assume this is the time to start breaking things out of void (loop) and into separate subroutines to allow void (loop) to execute faster?
Yes. Only call the subroutines(functions) as needed.
I think the solution is to not draw to the display every loop and only update it if I need to.
Definitely, same goes for all the functions
The buttons are currently on normal digital IO pins, not interrupt pins-so I think this is a case of attachInterrupt() being needed?
No, Once the loop() function is fixed polling the buttons with a debounce if necessary is preferred.
I like to think of the loop() as a dispatcher. Its only job is to know when to call in the other functions that do the real work.