Writing data to csv-file on SD-Card fails

Other Hardware Storage
1

Hi there,

I'm trying to log data to a csv-file on a SD-Card. When I open the file on my computer with an editor (tried different editors...) The file is corrupted. It contains all the data, some of the data or weird data, depending on how I wrote to the file. Generating the exact same content of the file is not generally repeatable. I tried different sketches, other people seemed to use successfully, and different libraries, but had no success so far.

First of all, some information about my project in general: I'd like to design a data logger to log the flight control data and accelerations of a remote controlled helicopter. Due to the higher writing speed, I log the data to EEPROM-Chips during the flight and copy the data to the SD-Card, afterwards. For very high resolutions, I can fly for only a couple of minutes. Therefore, I would like to land, store the data on the SD-Card , erase the EEPROM and take-off without restarting the Arduino. Thus, I need to close the file each time I finished writing to the SD-card.

Now about my issue: I would like to store some data (ints and floats) separated by ";" in a text-file. If I open the file on my PC, the editor shows random information and in the last row(s) of most files red highlighted \FF, @ and/or \00 several times. I suppose this is due to erroneous control characters. I reckon , file.close() should print a "end of file" to the file and open(filename) should remove it before printing new lines to the file. I tried to find some information by studying the SD-Library, but couldn't find anything. I couldn't find anyone else with this problem, so I assume there is a problem with my code or even more general (compiler, os, editor...).

At the end of the post, you can find some piece of example code. I chose the SD-Library provided by the Arduino-IDE, since I reckon this library should work most reliably. The sketch writes into several test files, using the functions file.print(), file.println() and file.write(). Furthermore I use "\r\n" and "\n" as control character for the end of a line. Reading the files on the serial terminal works fine. In the following, some examples of the output of some files, when I open them with gedit. The outputs of the other files is similar ot the examples given. If I open the file with nano (a terminal based editor), \00 is replaced by @. If I run the code several times, the number of \00 increases. test1.txt seems to be alright, if I open it after the first run. After a third run, it returns the output given in the third example, however the serial monitor returns the correct output.

Output test0.txt:

[Cached]
Size=9531
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00

Output test3.txt:

15751
kio_trash
Laptop
\00\00\00\00\00\00

btw: Laptop is the name of my Computer;)

Output test1.txt after third run:

1;2;3;
1;2;3;
1;2;3;
1;2;3;

1;2;3;
1;2;3;
1;2;3;
1;2;3;

1;2;3;
1;2;3;
1;2

Can anyone point me in the right direction, please?

2

And here the example code. I tried to explain my problem as short and precise as possible, but apparently, it took more than 9500 characters;) .

#include <SD.h>
#include <Arduino.h>

#define SD_CS_PIN 28

File myFile;

void setup()
{

  // Open serial communications
   Serial.begin(9600);

	Serial.println(F("Type any character to start"));
	while (!Serial.available()) {}

   Serial.println("Initialisation of SD card...");
   // On the Ethernet Shield, CS is pin 4. It's set as an output by default.
   // Note that even if it's not used as the CS pin, the hardware SS pin
   // (10 on most Arduino boards, 53 on the Mega) must be left as an output
   // or the SD library functions will not work.
    pinMode(53, OUTPUT);

   if (!SD.begin(SD_CS_PIN)) {
     Serial.println("   ...failed!");
     return;
   }
   Serial.println("   ...done.");

   Serial.println("");
   //==============================================================
   //test0
   // open the file. note that only one file can be open at a time,
   myFile = SD.open("test0.txt", FILE_WRITE);

   // if the file opened okay, write to it:
   if (myFile) {
     Serial.println("Writing to test0.txt...");
     myFile.print("1;2;3;\r\n");
		 myFile.print("1;2;3;\r\n");
		 myFile.print("1;2;3;\r\n");
		 myFile.print("1;2;3;\r\n");
		 myFile.print("\r\n");
     // close the file:
     myFile.close();
     Serial.println("   ...done.");
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test0.txt");
   }

   // re-open the file for reading:
   myFile = SD.open("test0.txt");
   if (myFile) {
     Serial.println("Read from test0.txt:");

     // read from the file until there's nothing else in it:
     while (myFile.available()) {
         Serial.write(myFile.read());
     }
     // close the file:
     myFile.close();
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test0.txt");
   }
   Serial.println("");
   //==============================================================
   //test1

   // open the file. note that only one file can be open at a time,
   // so you have to close this one before opening another.
   myFile = SD.open("test1.txt", FILE_WRITE);

   // if the file opened okay, write to it:
   if (myFile) {
     Serial.println("Writing to test1.txt...");
     myFile.print("1;2;3;\n");
		 myFile.print("1;2;3;\n");
		 myFile.print("1;2;3;\n");
		 myFile.print("1;2;3;\n");
		 myFile.print("\n");

     // close the file:
     myFile.close();
     Serial.println("done.");
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test1.txt");
   }

   // re-open the file for reading:
   myFile = SD.open("test1.txt");
   if (myFile) {
     Serial.println("Read from test1.txt:");

     // read from the file until there's nothing else in it:
     while (myFile.available()) {
         Serial.write(myFile.read());
     }
     // close the file:
     myFile.close();
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test1.txt");
   }
   Serial.println("");
   //==============================================================
   //test2
   // open the file. note that only one file can be open at a time,
   // so you have to close this one before opening another.
   myFile = SD.open("test2.txt", FILE_WRITE);

   // if the file opened okay, write to it:
   if (myFile) {
     Serial.println("Writing to test2.txt...");
     myFile.write("1;2;3;\r\n");
	 myFile.write("1;2;3;\r\n");
	 myFile.write("1;2;3;\r\n");
	 myFile.write("1;2;3;\r\n");
	 myFile.write("\r\n");
     // close the file:
     myFile.close();
     Serial.println("   ...done.");
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test2.txt");
   }

   // re-open the file for reading:
   myFile = SD.open("test2.txt");
   if (myFile) {
     Serial.println("Read from test2.txt:");

     // read from the file until there's nothing else in it:
     while (myFile.available()) {
         Serial.write(myFile.read());
     }
     // close the file:
     myFile.close();
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test2.txt");
   }
   Serial.println("");
   //==============================================================
   //test3
   // open the file. note that only one file can be open at a time,
   // so you have to close this one before opening another.
   myFile = SD.open("test3.txt", FILE_WRITE);

   // if the file opened okay, write to it:
   if (myFile) {
     Serial.println("Writing to test3.txt...");
     myFile.write("1;2;3;\n");
		 myFile.write("1;2;3;\n");
		 myFile.write("1;2;3;\n");
		 myFile.write("1;2;3;\n");
		 myFile.write("\n");
     // close the file:
     myFile.close();
     Serial.println("   ...done.");
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test3.txt");
   }

   // re-open the file for reading:
   myFile = SD.open("test3.txt");
   if (myFile) {
     Serial.println("Read from test3.txt:");

     // read from the file until there's nothing else in it:
     while (myFile.available()) {
         Serial.write(myFile.read());
     }
     // close the file:
     myFile.close();
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test3.txt");
   }
   Serial.println("");
   //==============================================================
   //test4
   // open the file. note that only one file can be open at a time,
   // so you have to close this one before opening another.
   myFile = SD.open("test4.txt", FILE_WRITE);

   // if the file opened okay, write to it:
   if (myFile) {
     Serial.println("Writing to test4.txt...");
     myFile.println("1;2;3;");
		 myFile.println("1;2;3;");
		 myFile.println("1;2;3;");
		 myFile.println("1;2;3;");
		 myFile.println("");
     // close the file:
     myFile.close();
     Serial.println("   ...done.");
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test4.txt");
   }

   // re-open the file for reading:
   myFile = SD.open("test4.txt");
   if (myFile) {
     Serial.println("Read from test4.txt:");

     // read from the file until there's nothing else in it:
     while (myFile.available()) {
         Serial.write(myFile.read());
     }
     // close the file:
     myFile.close();
   } else {
     // if the file didn't open, print an error:
     Serial.println("error opening test4.txt");
   }
}

void loop()
{
     // nothing happens after setup
}
3

You have a lot of string literals. You should wrap them ALL in using the F() macro. like you do the first one.

How much memory do you have available?
http://playground.arduino.cc/Code/AvailableMemory

4

Thanks for the hint. In my actual sketch I use the F() macro. I don't really care in the example. I always use a sketch like that to test functions before I actually implement them. This sketch is often rather messy;)

Apart from that, my external EEPROMS add 524288 byte to the memory of my mega:D

5

Is your EEPROM or any other device on the SPI bus? If you have any other SPI device put the chip select pin for the device in output mode and set the pin high.

Check your SD card for file system errors or reformat the SD using the SD Association's formatter SD Memory Card Formatter for Windows/Mac | SD Association.

I chose the SD-Library provided by the Arduino-IDE, since I reckon this library should work most reliably.

The standard library should work for you but it is just a wrapper for a very old version of SdFat I wrote five years ago and has a number of missing bug fixes.

One feature in newer versions of SdFat is a CRC check to rule out SPI bus errors. The Arduino has no hardware CRC so marginal SPI signals can cause this type problem. You can enable software CRC in newer versions of SdFat by editing SdFatConfig.h.

How fast do you need to log data? The new SdFat has an example that can log data records at 500 Hz with a time jitter of a few microseconds. A number of users log accelerometer plus other data using this example.

I also wrote an RamDisk file system that some people use with about a MB of FRAM or SRAM GitHub - greiman/RamDisk: RAM Disk library for Arduino.

Sound like your EEPROMS already work.

6

The EEPROM is on the same SPI-bus. The CS-pins are set as outputs and set high, when there is no communication. If my Arduino sends data to the SD-Card, is the task cycle stopped, or does it carry on? If it carries on, my datalogger might send data over the same SPI bus to the EEPROM, which could cause the problem.

I can't use the SD Association's formatter, since I'm running Linux. I did a filesystem check with a different tool, which turned out alright.

I had a look at the SdFat-Library to try to find the solution of the problem myselfe. I also tried to use SdFat without the wrapper. But unfortunately with no success.

In general, I'd like to log data with 10Hz, in extreme cases maybe 100Hz. I'm currently logging 64byte per sample, which might increase in the future. How much data can you log with 500Hz? A friend of mine recommended using EEPROMS, so I did. I'm pretty new to Arduino.

The EEPROMS work fine. I had some help writing the library, but it turned out pretty fancy:D

Edit: After thinking about what fat16lib worte after I posted the text above, I realised, that I should set the EEPROM Pins and deselect them by setting them HIGH, when I run my test routine on my board... The results after two runs look like as follows:

test0.txt:

1;2;3;
1;2;3;
1;2;3;
1;2;3;

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

test1.txt:

1;2;3;
1;2;3;
1;2;3;
1;2;3;

1;2;3;
1;2;3;
1;2;3;

test2.txt:

1;2;3;
1;2;3;
1;2;3;
1;2;3;

1;2;3;
1;2;3;
1;2;3;
1;2;3;

ÿÿÿÿÿÿÿÿÿÿ

test3.txt:

1;2;3;
1;2;3;
1;2;3;
1;2;3;

1;2;3;
1;2;3;
1;2;3;

test4.txt

[Cached]
Size=9739
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00

I'd say, this is an improvement, but still not the way it is supposed to turn out. And I still can't explain the missing lines, the \00 or the ÿ...

7

The old SdFat in SD.h may still have SPI sharing problems. It should be OK in your tests where you do not access the EEPROM.

To verify you have no SPI problems do the following.

Clone or download SdFat GitHub - greiman/SdFat: Arduino FAT16/FAT32 exFAT Library and install it.

Edit SdFatConfig.h and change this symbol from 0 to 2.

/**
 * To enable SD card CRC checking set USE_SD_CRC nonzero.
 *
 * Set USE_SD_CRC to 1 to use a smaller slower CRC-CCITT function.
 *
 * Set USE_SD_CRC to 2 to used a larger faster table driven CRC-CCITT function.
 */
#define USE_SD_CRC 0

Run the bench example https://github.com/greiman/SdFat/tree/master/SdFat/examples/bench.

This will verify SPI data transfers using the standard 16-bit SD CRC.

You can also format SD cards using the SdFormatter example https://github.com/greiman/SdFat/blob/master/SdFat/examples/SdFormatter/SdFormatter.ino.

Linux, OS X, and Windows do not conform to the SD standard. SdFormatter uses proper cluster sizes and alignment so file system boundaries match flash chip programming and erase boundaries.

8

I formated the card with your sketch and ran the bench.ino. Here's the result:

Free RAM: 6769
Type is FAT32
Card size: 3.95 GB (GB = 1E9 bytes)

Manufacturer ID: 0X1B
OEM ID: SM
Product: 00000
Version: 1.0
Serial number: 0X13776F02
Manufacturing date: 12/2012

File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
162.67,59520,2756,3141
158.27,59448,2756,3228
158.43,59792,2760,3225
158.29,59216,2752,3228
158.24,61840,2756,3229
158.56,59372,2760,3222
158.42,61804,2760,3225
158.62,59052,2760,3221
158.47,62016,2760,3224
158.72,61804,2760,3219

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
220.19,4872,2248,2319
220.12,4868,2248,2320
220.10,4872,2248,2320
220.09,4880,2248,2320
220.06,4868,2248,2320

Done

Is that good?

Which of those examples you provide with the library would you recommend me to take as reference for my data logger? Is there a way to find out or to make sure that the library doesn't access the SPI-bus anymore?

Thanks a lot for the advice!