I took a look at the code. You have SPCR = B01010011, and in the comments I see "clock = f/4". Maybe I misunderstand, but isn't that clock/128? Shouldn't SPR1 and SPR0 be set to 0 to get clock/4?
I changed both to 0, and enabled SPI2X in SPSR. On my best card, the writes are now 80ms per 512byte block!
Is this safe? Is there any error checking?