When you write to an SD card, the process is very slow. So, you actually write to a buffer, and when the buffer is full, it gets written to the card. Most writes then are fast, with only the occasional one that takes a relatively long time.
It is that 512 byte buffer which is using up your SRAM, and, no, you can't change the buffer size.