SD.begin() causes "slowness" when SD card is not present

I tried this using a combined TFT/SD-reader that work both with (the same) SPI, but leaving out the SD-card makes the display very slow... Other ideas?