Hi - thanks for your help. I'm happy to report I've got it working!
I took your suggestion to cut right back my code and slowly added things until it stopped working - turns out the problem was a conflict with the motor shield I'm using - it was making use of timer 1 and enabling both pins 9 and 10 for PWM output. Although the SD card is CS pin 4, pin 10 is the CS pin for the ethernet in my board and the PWM output was messing with the SPI somehow. It's a bit frustrating they would choose those particular pins as timer 1 is the only 16 bit timer and to have it clash with the ethernet is pretty frustrating. Anyway fortunately I'm only using 1 motor on the dual motor shield, so I was able to steal the bit of code from the Timer1 library to disable the PWM output for pin 10 (I only need pin 9 for the motor speed control). All is working now - I can leave the file open and just periodically flush the data to secure it on the SD card.