Use P-channel MOSFET on the VCC pin instead. Bring all control lines low to avoid powering the card thru the control lines.
You need to use a new version of SdFat to insure the card will sleep properly when idle. The old version of SdFat in the standard SD.h library has a bug that may prevent low power sleep.
Does the new SDFat.h library automatically enters into sleep mode when idle or do I need call a specific function?