I actually found someone on GitHub who by the looks of it tried to change the AudioZero library to just include SdFat but I'm not sure he actually got it working because when I tried it failed.
The reason I stored them in RAM is speed. If I had to grab the next one from the SD card it could add a lag to the operation. But in hindsight maybe you are right because I now see a similar project took that same approach.
And another thing I could do to improve my code is using an interrupt for the scrolling display... but that is why we have GitHub.