SD cards, are they worth the hassel....????

Hi! i am currently working on a project that requires the use of an SD card, and a RTC unit. I have been having some problems because the setup will lose connecting with the card after sometime, there does not seem to be any reason for this or any patten in the amount of time that it takes.. I have read almost every post that i can find on SD cards (there seems to be a lot) and have gotten some tips that seem to improve it.
But my main question is; are there people out there that are having long term success with writing and reading to/from SD cards? My whole project depends on me being a able to do this in a reliable way, it is also the only option that i have as i can not have it linked to a computer or cloud etc, so would be good to find out early in the game :slight_smile:

If you are interested or feel that you need more info here is a bit more...

The project will be running for months at a time logging a small electrical network, it will be getting power from that network so when the network is down (approx <1% of the time) the arduino will lose power as well. I will be reading some previously saved data at the start of each re-start and then writing about 18 3 digit numbers to the file 2-3 times a second, and start a new file for each week. I dont think that there is anything wrong with the part of the code that is working with the file as it is something that i have copied from an example on the net, but you can see the main parts of the code attached. :slight_smile:

https://www.ebay.com/itm/1-2-5-10PCS-SD-Card-Module-Slot-Socket-Reader-For-Arduino-ARM-MCU-Read-And-Write-/264036017585?var=&hash=item3d79c5e5b1

https://www.ebay.com/itm/Electric-DS3231-IIC-Module-DIY-RTC-Replace-Real-Time-Clock-Expansion-For-Arduino/392289529511?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2057872.m2749.l2649

Arduino Mega 2560 R3 ATmega2560-16AU

sketch_jun19a.ino (5.11 KB)

I had purchased one of these to use in a project. However, it was more hassle in my opinion then it was worth. I had many other ways for passing data between systems that I put it in my parts bin and have not looked at it for some time.

However, I see you are constrained as to what you can do and how you can get this data as such, it does not leave you many options.

Personaly, I love the RF24 Transceivers for moving data between Arduino and Raspberry Pi.

I also make use of MQTT when I can.

the SD module in link doesn't have logic level conversion. it is for boards with 3.3 V logic level

It should not disconnect if it is continually powered and vibration is not extreme. (The SD card in your helmet camera won't stay connected if you land on the helmet.)

There are better libraries than the standard SD.h which will allow you to reconnect to the card.

For a long-term logger, you should always close the file between writes.

OP's code...
Next time, use auto-format (CTL+T in the IDE) to prepare your code for listing.

//program for logging power consumption from 3 phase generators. 


// import library for clock
#include <DS3231.h>
// Init the DS3231 using the hardware interface
DS3231  rtc(SDA, SCL);

int debugSDcard = 1; // if this is anything other than "1" the SD card will not in initialize, but there is a couple of other places that i need to use "//" as this "function" does not remove everything. 

#include <SPI.h>
#include <SD.h> //Load SD library
const int chipSelect = 53; //chip select pin for the MicroSD Card Adapter
File file; // file object that is used to read and write data

int powerSource[12]; //where the power is coming from, i.e. gen set 1,2,3... trasnformer, battery-inverter.....

uint32_t maxTimeA = 0; // time logged in millis when the amps recorded its peak
uint32_t maxTimeV = 0; // time logged in millis when the amps recorded its peak
uint32_t diffTimeUL = 0; // difference between these two values
const float pi = 3.1416;

float Amps[4];
float Volts[3];
float PowerFactor[3];

uint32_t start_time;
uint32_t old_time = 0;

uint32_t counter = 0;// just for use while de-buggin, can be deleted later


float PSI = 0; // Power Supply Instant, the sum of volts*Amps for the 3 phases each time this code is run, needs to be set to once per second before the program can be used.
float PST; // Power Supply Total (sum of PSI)

// enableing the physical removal of the SD card from the system ---------------------------
int i; //for the while loop that i am testing for writing data to the SD card
const int SDCardRe = 8; 
const int SDCardFail = 3;
 
// Setup for Hertz measurement 
const int HzPin = 7;
unsigned long HzVal;
unsigned long durationHigh; //need to use the duration for the wave above and below as there is some noise in the system which does not give me a perfect sin wave.
unsigned long durationLow;

// Setup for measureing of Amps in the system -----------------------------------------------

// Phase 1 current setup
int sensorAmps[4] = {A0, A1, A2, A3}; // Analog input pin that sensor is attached to
int sensorVolts[3] = {A4, A5, A6}; // Analog input pin that sensor is attached to 
 
void setup() 
 {
   Serial.begin(9600); 

//-----------------------------------------------------------------------------------------------

// setting up the SD card system

if (debugSDcard ==1){
  pinMode(chipSelect, OUTPUT); // chip select pin must be set to OUTPUT mode
  if (!SD.begin(chipSelect)) { // Initialize SD card
    Serial.println("Could not initialize SD card."); // if return value is false, something went wrong.
  }
  
  if (SD.exists("file.csv")) { // if "file.csv" exists, file will be deleted
    Serial.println("File exists.");
    if (SD.remove("file.csv") == true) {
      Serial.println("Successfully removed file.");
      file.print("all good at this stage");
    } else {
      Serial.println("Could not removed file.");
    }
  }
}

  //-----------------------------------------------------------------------------------------------
  // setting up the clock and printing it to the file only at the start of the file
  rtc.begin();

if (debugSDcard ==1){  
    file = SD.open("file.csv", FILE_WRITE); // open "file.txt" to write data
    if (file) {
  
  
    } else {
    Serial.println("Could not open file (writing 1).");
  }
}

 }

void loop() {


if (digitalRead(8)==LOW){

if (debugSDcard ==1){
file = SD.open("file.csv", FILE_WRITE); // open "file.txt" to write data
}
   if (file) {

   Time t = rtc.getTime();
   start_time = t.sec;
     
   i = 0; 
   //start_time = millis(); //as i only want this loop to run 2 times a second i have added "this 498" millis to the code. (498 and not 500 because of aculitive error)
   
   while  ((i < 100) && (digitalRead(SDCardRe) == LOW) && (old_time != start_time)) //closing and opening the file for every 100 runs so it saves
   {
         
   old_time = start_time;
      
   i =i+1;
  

    counter = counter+1;  // can be removed at after all is wroking well
   
   /*this next bit is not perfect as if the Hz is zero as it sometimes can because of wrong measurment then it 
    * will screw up that one measurement but i dont think that this is much of a problem. 
    */

  if (counter < 101){
    Serial.print(counter), Serial.print(";");
    Serial.print(rtc.getTimeStr()), Serial.print(";");
    Serial.print(HzVal), Serial.print(";");
    for (int i = 0; i < 4; i++) Serial.print(Amps[i],0), Serial.print(";");
    for (int i = 0; i < 3; i++) Serial.print(Volts[i],0), Serial.print(";");
    for (int i = 0; i < 3; i++) Serial.print(PowerFactor[i]), Serial.print(";");
    Serial.print(PST,0), Serial.print(";");
    Serial.println();    

  }

    file.print(counter), file.print(";");
    file.print(rtc.getTimeStr()), file.print(";");
    file.print(HzVal), file.print(";");
    for (int i = 0; i < 4; i++) file.print(Amps[i],0), file.print(";");
    for (int i = 0; i < 3; i++) file.print(Volts[i],0), file.print(";");
    for (int i = 0; i < 3; i++) file.print(PowerFactor[i]), file.print(";");
    file.print(PST,0), file.print(";");
    file.println();

   
   }
   }
   
   file.close(); // close file
    } else {

    Serial.println("Could not open file (writing 2).");
    }
}

Look & Learn

I use the no-hassle Sparkfun OpenLog and am quite happy with it. No programming or libraries required.

It is cheap, extremely reliable and simply records whatever text comes over the serial port, automatically opens files for writing, etc.. Just pop into your Android phone or uSD slot to download the data.

3.3 V only I/O, so if you are using it with a 5V Arduino, at the very least put a 2.2K resistor in series with the TX pin.

I agree with other comments that you may have the wrong kind of microSD module. The one in your link appears to be for use with 3.3V MCUs, not the 5V Mega. You need one with a voltage translator chip. I think one like this would be better:

But aside from that, I can't claim to have long-term success using SD cards with Arduinos, but I did write an SD card bootloader for another processor class, and found that there are two basic requirements for the software to get right. The first is initializing and communicating with the card, and the second is dealing with the FAT file system. Cards from different manufacturers may behave slightly differently, and your library needs to deal properly with those variations. Anyway, not all libraries and code examples are created equal, so your best bet may be to try different ones, for both SD and FAT, to see if they work better. But so far as I know, there's no inherent flakiness in SD cards that prevents you from getting reliable performance. They are used in millions of cameras, smart phones and tablets everyday, and you don't see lots of complaints.

Thanks a lot for the replies!! Yes, well i was not aware that the card was not getting 3.3V as i thought that because it had the 3.3v pin on it then everything was ok, never thought about the other signals coming from the MISO etc pins not being 3.3v. i will forsure fix this now and see if it is helping!! :slight_smile:

I use micro SD cards quite a bit on 3.3V Arduinos.

Some micro SD cards just refuse to initialise, but the ones that do, then work reliably.

I've not used the modules or SD libraries, outside of the wire library for the SPI bus. I've never used an SD card in the proprietary format (higher speed they claim) just in the non-proprietary SPI method. It works like any memory chip that can be attached to an SPI bus. I haven't used it with a traditional file system or other OS components, just raw memory. When I need to access the data externally, the ATmega reads the SD and dumps ascii out the serial port, which is usually formatted into a CSV. You definitely want to control read and write, so as not to corrupt the memory, and setup the SPI correctly, so as not to have crosstalk with other SPI components. Outside of those caveats, it's very reliable. Yes, it operates at 3.3 volts and while there are lots of folks who will argue that a 5 volt ATmega can talk to and listen to a 3.3 volt data source, I will only say that it's out of spec, so maybe I'm over-engineering it, but I always have appropriate logic level translation in the circuit.

Perehama:
there are lots of folks who will argue that a 5 volt ATmega can talk to and listen to a 3.3 volt data source, I will only say that it's out of spec,

see the datasheet. ATmega will read the pin HIGH from 0.6 * Vcc, so at 5 V Vcc the threshold is 3 V.

I agree with jremington , I use the open log but made my own using the pro mini and one of these off e-bay.

https://www.ebay.co.uk/itm/Arduino-Micro-SD-Card-Mini-TF-Card-Reader-Module-SPI-Interface-Raspberry-PI-C29/283448398768?var=&hash=item41fed753b0&enc=AQADAAADAFjVrDbVsZ8oH%2F8PNHtt9VX4%2Fw7FZcmMuqsX8uaFEduV1RndpWkAFfczWueUX7shdSsS6MzRzOPPEQDDnu9dwE3l%2B2qJaTOcd2113KhyqtRxdZxgIma9ioPb9IbbucHxULwuw53FqXjy9UzjyBbz0kBdZ1LVD5WErek6cY4fkO5K8pEWaveT9ClBQJpWOX24jLSYoaKYCAEKkRnmllhn%2FYcOdw%2F3MVyP7CNpOOEEOEFct6Novl7IPpAvKj1CZmowayMmz58%2FhQMjxaBvM2utOsXLLJaBHMVafFU%2BB3MlKKs4HxUoGKsqHKqcPMKT9d9HuqOjbVhGxmGrl8ymESScKAi1B3EE0LHeVQ8Sk4PmFiw9eKu%2F3ZlgZgP%2FR8Ys74AHW3PErIT2HyaL%2FPVtPpgmGoAOceB9ra2NqQcNmmclTrweA5%2FV7oxitx2fnNWE6TdvXo8Ruy50d5Qugry%2FWDxn5SamqUvQ%2Bxgvi%2BCnLhRUOip62idALPU3hYCnjWH5oK46if3OKYIno2LRpOm9lVjhOPJTmEoTpOwBQuIn%2FgkWeI6oj9VtzHlMUF6ipsWLpvDP0N6D7xzkEY8eJb1ptht42zh8KAZ3MVYFJIxZMIfFj%2BA6GY0kik7G7QtJVxMviCxpyzUUBkfY3xzmcxg5oD68iRIgy2APrNpfCDtftbFVmPH0VzfS1J7C8swNSGjNEVXzBAVnw8E0%2FlN%2BXuAy9wJEKG8caKz%2FcPTLv5AfzzEnX3RIZscaSBevxU0f6qephVe7XjPEcMdTcP%2FKv1%2F9M%2FpwOP1ps0sxV1UaT0UEwNMS%2FgSqxKADdfjT7MxiQ9kOfDE0Z8eUljF%2BpcD9xikUHxuqBw1K2XPmUt7e4fUyj7WuG3q7IhWc7wNBKWjRe9W00rXpSnnkbmb3ir%2FQirdbF7XgXA83PHpN6gYyMciualGQF9PJzo5k%2Fqmm3zMub80vCLCj7lBcboT%2Faz3Seg8FaBtJr2aCvxvMXEZH1WTjxi9mazl2brZ%2FVpeuKr0UnXbrh9rYTA%3D%3D&checksum=283448398768b460dd61f04d4c3b965f9b70c0722238&frcectupt=true

Take all the hassle out on converting the voltage's from 5V to 3V. I've been suing mine for some time now and not had any issues with it.

Juraj:
see the datasheet. ATmega will read the pin HIGH from 0.6 * Vcc, so at 5 V Vcc the threshold is 3 V.

This is the calculation for Vih minimum for the ATmega, and as long as Voh minimum for the 3.3 logic is above 3 volts, it'll work, but there are a number of 3.3 devices, such as these SD cards from Kingston, that state that Voh minimum with respect to vcc (3.3) falls below 3 volts. Also, most SD cards run on 2.7 to 3.6 volts, so like I said, it's something someone's going to contest, but it is out of spec.