Awful problem with SD Datalogger tutorial code

I'm working on a project that uses the SD card to store both bitmaps for use on the tft display, and log files for recording sensor values. The bitmaps open with no problem, and are displayed on the TFT using the Adafruit GFX library.

However, when I go to use the log file, following the "Datalogger" tutorial, the log file cannot be opened. This happens even when I run the Datalogger example by itself. The card initializes, but the file won't open.

I'm using the line of code straight from the tutorial:

  File dataFile = SD.open("LOG/datalog.txt", FILE_WRITE);

There is no problem writing to the log file in setup, but it will not write in loop.

Upon opening the log directory on the card after letting, it seems that there are multiple files with corrupted names, dates, and sizes. The log file does exist too, with the line wrote during setup.

Any idea about what's going on?

Sorry, I was editing the post to reflect that I did format the card, and while the corrupted files don't return when I run the sketch, the log file still can't open in the loop (but it does in setup with identical code).

I'll try formatting again with the utility you suggested.

So I tried formatting in FAT 16 instead of FAT 32, and now my sketch deletes the bitmaps from the card, and still doesn't write to the log file in the loop. Supposedly the Adafruit GFX code can read both 16 and 32, but oh well...?

I changed it back to 32 and I'm back where I started.

The problem is not likely format but you should not use OS utilities to format SD cards. The SD association formatter formats them correctly.

2 GB SD and smaller cards should be FAT16 with the correct cluster size and layout. SDHC 4-32 GB cards should be FAT32 with the correct file system layout.

Double check that the open in setup is really identical to the open in loop.

Make sure the file is closed before reopening it.

A major reason for open failing is lack of memory. open() tries to allocate memory and if memory is low, it will fail.

I'm on Ubuntu so I used mkdosfs, but I'll try the SD association formatter on a windows machine in a bit. 4GB card so FAT32 it is.

Open in setup is identical to the open in loop. I run SD.begin in setup; I don't need to run it in loop also, do I?

Looks like all opens are correctly paired with closes, so nothing should get left open.

How can I tell if open is failing due to a lack of memory? I'm on a Mega 2560. There is no problem opening a 58kb bitmap, but always a problem opening the log (or creating it if it doesn't exist).

You only need SD.begin() in setup.

Memory is not likely a problem on a Mega.

Is the path a literal in open()? If not is the path variable correct in loop?

The fact that an SD needed to be reformatted may mean you are writing over memory. Writing over the SD cache can cause this type problem.

Paths are literals in both setup and loop.

The formatting problem hasn't happened since, so I think it was just due to something goofy I did in the past.

Wow, I did some research for the "LC Studio" SD breakout that I'm using, and while it has a 5 to 3.3v voltage converter for power, the data lines are not converted.

It would then make sense to me that I could read from the card, as the Mega sees a 3.3v as HIGH, but not write to the card, as the data signals are coming in at 5v. However, there is no problem when writing to the card in setup.

Could this be my problem? Either way, I'll test it out with a SparkFun level converter that I have...

Update: Finally got the level shifter in line with the SD breakout, but that wasn't the problem.

After I began commenting out certain portions of the loop, I found the offending code.

    digitalWrite(_oe, LOW);
    
    TCCR1B = (TCCR1B & 0xF8) | 1; // generates the MCKL signal 
    analogWrite (_clock, 128); 
  
    resetsensor () ;// resets the sensor - caution: afterwards mode = SPI_MODE0! 
    
    long c1 = 2607; 
    long c2 = 4696;
    long c3 = 367;
    long c4 = 224; 
    long c5 = 2289; 
    long c6 = 52;
    
      resetsensor () ;// resets the sensor 
  
    // Temperature: 
    unsigned int tempMSB = 0; // first byte of value 
    unsigned int tempLSB = 0; // last byte of value 
    unsigned int D2 = 0; 
    SPI.transfer (0x0F); //  send first byte of command to get temperature value 
    SPI.transfer (0x20); // send second byte of command to get temperature value 
    delay (35); // wait for conversion end 
    SPI.setDataMode (SPI_MODE1); // change mode in order to listen 
    tempMSB = SPI.transfer (0x00); // send dummy byte to read first byte of value 
    tempMSB = tempMSB << 8; // shift first byte 
    tempLSB = SPI.transfer (0x00); // send dummy byte to read second byte of value 
    D2 = tempMSB | tempLSB; // combine first and second byte of value 
    //Serial.print ("Temperature raw ="); 
    //Serial.println (D2); // voilá! 
  
    resetsensor () ;// resets the sensor 
  
    // Pressure: 
    unsigned int presMSB = 0; // first byte of value 
    unsigned int presLSB = 0; // last byte of value 
    unsigned int D1 = 0; 
    SPI.transfer (0x0F); // send first byte of command to get pressure value 
    SPI.transfer (0x40); // send second byte of command to get pressure value 
    delay (35); // wait for conversion end 
    SPI.setDataMode (SPI_MODE1); // change mode in order to listen 
    presMSB = SPI.transfer (0x00); // send dummy byte to read first byte of value 
    presMSB = presMSB << 8; // shift first byte 
    presLSB = SPI.transfer (0x00); // send dummy byte to read second byte of value 
    D1 = presMSB | presLSB; // combine first and second byte of value 
    //Serial.print ("Pressure raw ="); 
    //Serial.println (D1); 
  
    // Calculation of the real values ??by means of the calibration factors and the maths 
    //  In the datasheet.  const MUST be long 
    const long UT1 = (c5 << 3) + 10000; 
    const long dT = D2 - UT1; 
    const double TEMP = 200 + ((dT * (c6 + 100)) >> 11); 
    const long OFF = c2 + (((c4 - 250) * dT) >> 12) + 10000; 
    const long SENS = (c1 / 2) + (((c3 + 200) * dT) >> 13) + 3000; 
    long PCOMP = (SENS * (D1 - OFF) >> 12) + 1000; 
    double TEMPREAL = TEMP/10; 


    _atmo = PCOMP; 
    Serial.print("PCOMP:  ");
    Serial.println(PCOMP);
    digitalWrite(_oe, HIGH);

It's somewhere in the code I'm using to access values of a pressure sensor. The sensor doesn't have a CS pin, so I'm running it through a 3 state buffer and using the Output Enable pin of that as a sort of CS pin.

Any idea on how to make the code less offensive to the SD card? I can try posting in the SPI forum too...

Came up with a fix.

First, after every call to the sensor, I change the SPI mode back to 0.

Second, I changed the clock rate to 4Mhz instead of 16.

Edit: Why the Datalogger tutorial sketch didn't work, I have no idea.

The Arduino SD.h library does not setup SPI mode before using the bus. SD.h is based an old version of SdFat I wrote in 2009. It has not had updates or bug fixes.

Newer versions of SdFat set SPI mode and speed before accessing the SD card.

SD.h also has poor examples. The datalogger example opens and closes the file each time data is logged.

Opening a file at the end can take a very long time depending on where the file is in the directory and how big the file is.

A better plan is to open the file once in setup() and call flush() in SD.h or sync() in SdFat after writing data. SD.h flush() just calls SdFat sync(). sync() write all cached data to the SD so you won't lose data on a reset or power down.

close() just calls sync() and sets a file closed bit in the file descriptor.

So you suggest I use the newest version of SdFat? :slight_smile:

First, what brand/model SD card are you using? Most users can't even get their SD card to initialize in the LC Studio module.

Using the newest SdFat without the SD.h wrapper will probably be smaller, faster, and have five years of bug fixes. Faster speed may be useful for bitmaps.

The Arduino company's SD.h wrapper was intended to make SdFat easier by changing the API and limiting access to some features.

For example SD.h runs at 4 MHz while SdFat has clock divider as the second parameter of begin().

Here is a read/write benchmark for SD.h with large 512 byte reads and writes using an industrial 1 GB SD.

File size 5MB
Buffer size 512 bytes
Starting write test. Please wait up to a minute
Write 284.46 KB/sec
Maximum latency: 12084 usec, Minimum Latency: 1736 usec, Avg Latency: 1793 usec

Starting read test. Please wait up to a minute
Read 319.06 KB/sec
Maximum latency: 3196 usec, Minimum Latency: 1588 usec, Avg Latency: 1598 usec

Here is the same benchmark with the newest SdFat:

File size 5MB
Buffer size 512 bytes
Starting write test. Please wait up to a minute
Write 509.76 KB/sec
Maximum latency: 19324 usec, Minimum Latency: 936 usec, Avg Latency: 998 usec

Starting read test. Please wait up to a minute
Read 553.74 KB/sec
Maximum latency: 1844 usec, Minimum Latency: 908 usec, Avg Latency: 919 usec

Its a Class 4 PNY 4GB SDHC that seems to work fine. And don't forget that I had it hooked up to 5v data lines for a while too.