I think I figured it out, it seems to work now. These are the things I did, each of them may or not have contributed to it working.
- Used the SdFat at GitHub - greiman/SdFat: Arduino FAT16/FAT32 exFAT Library, not the original SD library
- Used chipSelect = 10
- Used sd.begin(chipSelect, SPI_HALF_SPEED), begin and not cardbegin or any other method