That does seem slow.
How're you writing to the DAC?
If you figure an arduino with 16,000,000 clocks/second,
needs 8 clocks to read a byte serially in a burst mode,
so you're down to 2,000,000 reads/second,
and 8 clocks to write it out in a burst mode,
so you're down to 250,000 reads/writes/second,
and it takes 100 clocks for the instructions to move the data from input register to output register and other stuff,
so you're down to 2,500 reads/writes/second.
Doesn't leave much room for quality sound does it.
Doesn't sound like SD card with serial SPI would do much better.
If you could do byte reads/writes you could get back to ~40,000 reads/writes per seconds.
Or go with external hardware - load start address into a counter, have EEPROM data bus be same as DAC bus, address clock to counter be the data clock into the DAC kind of thing.