By default, when you open a file for writing, it will truncate the file and you start from the beginning.
Try changing
File logfile = SD.open("logger.csv", FILE_WRITE);
to
File logfile = SD.open("logger.csv", FILE_WRITE | O_APPEND);
and see if that helps.