Incomplete data write to SD carrd

Other Hardware Storage
1

Hello,
I'm looking for a little guidance on what might I'm doing wrong when I try to write strings to an SD card.
I'm aware that strings are not necessarily the most efficient way to store information, and I'm trying to learn how to convert the data types to character arrays. I'd request that people's replies be without some of the disparagement I've seen on other forum questions regarding strings, and perhaps some advice on casting floats, ints etc as character arrays (or anything else more appropriate) could be substituted.

I have a SD card writer that I'm communicating with an Arduino Uno, with

// ** GND
// ** Vcc 3.3 V
// ** CS - pin 4
// ** MOSI - pin 11
// ** CLK - pin 13
// ** MISO - pin 12

I'm attempting to write GPS, tiltmeter and magnetometer data to the card

I can write the data I need to both the Serial Monitor with

Serial.println(gpsString);
Serial.println(tiltString);
Serial.println(magString);

I've built the strings just by concatenating floats and ints casts as strings. An example output of what I see on the serial monitor (without trying to write to the SD card) is:

20:5:24:14:20:28.0 sxxx.xxxxxxxxx:xxx.xxxxxxxxx
-5 -79 17
1614 6509 -1

(the s and x are actual numbers that I've removed. They are my lat and long)

However, the call I'm using to write to the SD card seems to be corrupting the strings somehow. This call:

File dataFile = SD.open("datalog.txt", FILE_WRITE);
if (dataFile) {
dataFile.println(gpsString); //note that GPS use UTC time.
dataFile.println(tiltString);
dataFile.println(magString); 
dataFile.close();
}

or even just

File dataFile = SD.open("datalog.txt", FILE_WRITE);
dataFile.close();

Will produce return the lines:

20:5:24:14:16:19.0 
-4 -81 18
1618 6507 -1

i.e. without the lat and long values written, though the date and time are - even though I've cast them all as strings - i. e.

String gpsString = String(fix.dateTime.year) + ":" + String(fix.dateTime.month) + ":" + String(fix.dateTime.date)+":" ;
gpsString += String(fix.dateTime.hours) + ":" + String(fix.dateTime.minutes) + ":" +String(fix.dateTime.seconds);
gpsString += "." + String(fix.dateTime_ms())+" ";
gpsString += String(fix.latitude(),9) + ":" + String(fix.longitude(),9)+" ";

When I try to write to the SD card, the data saved to the cars has the same content as above (i.e. , GPS data including date time without lat, long, Tiltmeter and magnetometer are all saved. The only data that is absent is the lat and long.

I'm unsure, but I suspect I'm not using strings or variable types properly - and if that is the case, i'm not sure how to solve the problem - e.g. converting float or int variables into character arrays. Either way, does anyone have any thoughts on my problem?

2

Sorry, meant to post to storage forum.. mods advised..

3

Before you go any further I suggest that you distinguish clearly between Strings and strings in your post. You appear to know know that there is a difference, but understanding the problem would be easier if you wrote String when you mean String rather than writing string when you mean String. I assume that throughout your initial post you are referring to a problem with Strings

How are you reading the data stored on the SD card ?

4

Thanks for your response Bob - and thanks also the heads up - I'll be more careful in the future. As you say, I'm talking about Strings, though I believe it's correct to say that string arrays are more efficient and the recommended implementation -and I'm looking into that now too.

I'm reading the data stored on the SD card with external python software, which will do some processing, smoothing etc. The exact format of the data isnt important, as long as it's all there and readable by python.

Thanks again,
EM.

5

Would it be possible to post the complete sketch instead of snippets; if too big, attach it.

One thing that I consider advisable is to add an else to if (dataFile) so you're not blind if opening the file fails (although probably not the cause of the problem).

6

Many thanks for your suggestions Sterretje, and the offer to review the code. I've been working on it to try to reduce the number of String objects, and replace them with character arrays so it's a little different to the original snippets above - however the problem still persists in that the request to print to serial monitor for both lat and long at the same time, corrupts the string. Printing, for example lat, without printing long is successful, as is printing long without printing lat. printing both lat and long will print only lat, and questionmarks for long.

Again, thanks for your help.
EM.

adc_tilt_sd_GPS_rebuild_4.ino.ino (5.43 KB)

7

OP's code

#include <NMEAGPS.h>
#include <NeoSWSerial.h>
#include <Wire.h>  // include wire communciations library
#include <MCP342x.h> // inlude ADC library 
#include <MMA_7455.h> //Include the MMA_7455 library
#include <SPI.h> // SPI library (for SD card)
#include <SD.h>  // SD card read library

// onboard units
// SDCARD
// OLED
// TILT
// ADC - > FGM
// GPS

// -- SDA, SCL, gnd, 5v (lower module ADC, OLED, TILT )
// -- gnd, 5v, SDA, SCL ->  OLED
// -- gnd, 7, 8, 5V  -> GPS
// -- 3.3v, 4, 11,12,13, gnd-> SDCARD


// -------------------------- set up the SD card drive parameters--------------------------------
// ** GND
// ** Vcc 3.3 V ###IMPORTANT### 
// ** CS - pin 4 (for MKRZero SD: SDCARD_SS_PIN)
// ** MOSI - pin 11
// ** CLK - pin 13
// ** MISO - pin 12
Sd2Card card;
SdVolume volume;
SdFile root;

// -------------------------- set up the GPS parameters--------------------------------
NeoSWSerial gpsPort(7, 8); ////GPS RX= Ard pin 8, GPS TX=Ard pin 7 
NMEAGPS GPS;

// -------------------------- set up the ADC parameters--------------------------------
MMA_7455 mySensor = MMA_7455(); //Make an instance of MMA_7455 NEED THIS FOR ADC
uint8_t adc_address = 0x6E; // 0x68 is the default address for all MCP342x devices

MCP342x adc = MCP342x(adc_address);

// Resolution is 12, 14, 16,or 18; gain is 1, 2, 4, or 8.
MCP342x::Config config(MCP342x::channel1, MCP342x::oneShot,MCP342x::resolution16, MCP342x::gain1);
MCP342x::Config status;// Configuration/status read back from the ADC

void setup()
{
  Serial.begin(115200);
  gpsPort.begin(9600);
  
  
//+++++++++++++++++++++++  SET up ADC  ++++++++++++++++++++ 
  MCP342x::generalCallReset();
  delay(1); // MC342x needs 300us to settle
  
 // Check ADC present
  Wire.requestFrom(adc_address, (uint8_t)1);
  if (!Wire.available()) {
    Serial.print("No adc device found at address ");
    Serial.println(adc_address, HEX);
    while (1);
  } else {Serial.println("Found ADC");}


 ///+++++++++++++++++++++++  SET up TILT sensor  ++++++++++++++++++++ 
  uint8_t tilt_address = 0x1D; //address for tilt is at 0x1D, make sure two addresses are different.
  mySensor.initSensitivity(2); // set up  for "Tilt" measurements: 2 = 2g, 4 = 4g, 8 = 8g
  Wire.begin();
    
  // Check TILT present
  Wire.requestFrom(tilt_address, (uint8_t)1);
  if (!Wire.available()) {
    Serial.print("No tilt device found at address ");
    Serial.println(tilt_address, HEX);
    while (1);
  }else {Serial.println("Found TILT");}

 ///+++++++++++++++++++++++  SET up SD card drive ++++++++++++++++++++ 
  const int chipSelect = 4;  // this is used to signal comms. to SD 
  Serial.print("Initializing SD card...");

  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    while (1);
  }else {Serial.println("card initialized.");}

 ///+++++++++++++++++++++++  SET up GPS ++++++++++++++++++++ 
  Serial.print("Initialising GPS - please wait.");
  if (GPS.available( gpsPort )){
    gps_fix fix = GPS.read();
    if (fix.valid.location) {Serial.println("..GPS activated and operating");}}
}

  
void loop() {
 long magX, magY, magZ;
 char gpsString[20];
 char magString [10]= "";
 char  tiltString[10] = "";
 uint8_t dummyRead;
 
  if (GPS.available( gpsPort )) {
    gps_fix fix = GPS.read();
    if (fix.valid.location) {
        String yr = String(fix.dateTime.year)+" ", mnth = String(fix.dateTime.month)+" ", dte = String(fix.dateTime.date)+" ";
        String hrs = String(fix.dateTime.hours)+":", mins = String(fix.dateTime.minutes)+":", sec = String(fix.dateTime.seconds)+".";
        String ms = String(fix.dateTime_ms())+" ",lat = String(fix.latitude(),9)+" ",lon = String(fix.longitude(),9)+" ";
        strcpy(gpsString,yr.c_str());strcat(gpsString,mnth.c_str());strcat(gpsString,dte.c_str());
        strcat(gpsString,hrs.c_str());strcat(gpsString,mins.c_str());strcat(gpsString,sec.c_str());strcat(gpsString,ms.c_str());
        strcat(gpsString,lon.c_str());strcat(gpsString,lon.c_str());

        char tiltX = mySensor.readAxis('x'); //Read out the 'x' Axis
        char tiltY = mySensor.readAxis('y'); //Read out the 'y' Axis
        char tiltZ = mySensor.readAxis('z'); //Read out the 'z' Axis
        strcpy(tiltString,tiltX); strcpy(tiltString,tiltY); strcpy(tiltString,tiltZ);

        dummyRead = adc.convertAndRead(MCP342x::channel1, MCP342x::oneShot,
                 MCP342x::resolution14, MCP342x::gain1,10, magX, status);String magX_S=String(magX);
        dummyRead = adc.convertAndRead(MCP342x::channel3, MCP342x::oneShot,
                 MCP342x::resolution14, MCP342x::gain1,10, magY, status);String magY_S=String(magY);
        dummyRead = adc.convertAndRead(MCP342x::channel4, MCP342x::oneShot,
                 MCP342x::resolution14, MCP342x::gain1,10, magZ, status);String magZ_S=String(magZ);
        strcpy(magString,magX_S.c_str()); strcpy(magString,magY_S.c_str()); strcpy(magString,magZ_S.c_str());
      
      Serial.println(gpsString); //note that GPS use UTC time.
      Serial.println(tiltString);
      Serial.println(magString); 
/*
      File dataFile = SD.open("datalog.txt", FILE_WRITE);
        if (dataFile) {
        dataFile.println(gpsString); //note that GPS use UTC time.
        dataFile.println(tiltString);
        dataFile.println(magString); 
        dataFile.close();
        }
*/
      }
    }
}
8

Some notes (not solving your problem).

Wire.begin() should be, to my knowledge, the first thing before you call any other Wire functions. You currently have a Wire.requestFrom() before it.

Your code still has String objects; converting them to c-strings does not solve that :wink:
Below uses sprintf (some people prefer snprintf for safety) to put the date/time in gpsString without the use of string objects)

      //                  yyyyMM  dd  hh  mm  ss  ms
      sprintf(gpsString, "%04d%02d%02d%02d%02d%02d%03d",
              fix.dateTime.year, fix.dateTime.month, fix.dateTime.date,
              fix.dateTime.hours, fix.dateTime.minutes, fix.dateTime.seconds, fix.dateTime_ms());

Your gpsString is too small; make it 40 or even more for testing. How long are lat and lon; are they floats? I have no experience with GPS. If they are floats, you can not straight way put them in the above; you need dtostrf() to convert to text first,

The format that you want to achieve looks like
yyyyMMddHHmmssfff plus lon and lat; that first part is already 17 characters.

9

Thanks again for your help and tips - I've made the changes you suggest.

I did a little more sluthing to find out what might be going on - After the call to sprintf, it seems I cant open and write to the SD card/file any longer.

Commenting out the sprintf call means I can open/write to the SD card again.

It seems weird to me that those are connected, I'm copying my main code here, in case anyone has some thoughts.

void loop() {
  
 long magX, magY, magZ;
 char gpsString[65];
 uint8_t dummyRead;
 
// Check new data from GPS 
  if (GPS.available( gpsPort )) {
    gps_fix fix = GPS.read();
    if (fix.valid.location) {

// Get lat lon (double) conv. to string
     char lat[14],lon[14];
     dtostrf(fix.latitude(),13, 9, lat);
     dtostrf(fix.longitude(),13, 9, lon);

// Get magnetometer, x, y, z
     dummyRead = adc.convertAndRead(MCP342x::channel1, MCP342x::oneShot,
              MCP342x::resolution14, MCP342x::gain1,10, magX, status);
     dummyRead = adc.convertAndRead(MCP342x::channel3, MCP342x::oneShot,
              MCP342x::resolution14, MCP342x::gain1,10, magY, status);
     dummyRead = adc.convertAndRead(MCP342x::channel4, MCP342x::oneShot,
              MCP342x::resolution14, MCP342x::gain1,10, magZ, status);
// build the output string
     //format yy mm dd hr:mn:sc:mss ----------lat ----------lon Tix Tiy Tiz --------MX --------MY --------MZ
     sprintf(gpsString, "%2d %2d %2d %2d:%2d:%2d.%03d %13s %13s %3d %3d %3d %10lu %10lu %10lu",
     fix.dateTime.year, fix.dateTime.month, fix.dateTime.date,                          // date
     fix.dateTime.hours, fix.dateTime.minutes, fix.dateTime.seconds, fix.dateTime_ms(), //time
     lat,lon,                                                                           //lat, long
     mySensor.readAxis('x'), mySensor.readAxis('y'), mySensor.readAxis('z'),            // read & add in the tilt
     magX, magY, magZ);                                                                 // Magnetometer.

     
//Write out ot SD card
//testing follows 
      String dataString = "blah3";
      File dataFile = SD.open("datalog.txt", FILE_WRITE);
    
      if (dataFile) {
        Serial.println("inside file");
        dataFile.println(dataString);
        dataFile.close();
          } else {  
            Serial.println("Error opening file");
          }



      } // if fix.valid.location
    } // if GPS.available
}
  • just to iterate, right now, this setup produces "error opening file" written to the monitor. However, when i comment out the block following "//build the output string" (i.e. dont build the string), rerunning produces "inside file" to be written to the monitor instead.

thank you so much for your help so far.

10

Your gpsString is too small to hold all that; I think that 100 will be enough but not 100% sure; you'll have to do the counting.

For playing it safe:

      snprintf(gpsString, sizeof(gpsString) -1, "%2d %2d %2d %2d:%2d:%2d.%03d %13s %13s %3d %3d %3d %10lu %10lu %10lu",
              fix.dateTime.year, fix.dateTime.month, fix.dateTime.date,                          // date
              fix.dateTime.hours, fix.dateTime.minutes, fix.dateTime.seconds, fix.dateTime_ms(), //time
              lat, lon,                                                                          //lat, long
              mySensor.readAxis('x'), mySensor.readAxis('y'), mySensor.readAxis('z'),            // read & add in the tilt
              magX, magY, magZ);                                                                 // Magnetometer.

It will truncate the data if gpsBuffer is too small but should stop the problem.

Note: %2d will print a minimum of two characters; the year 2020 is 4 digits/characters so %2d will still print 2020 (if the gps indeed gives the full year).

11

N.B.: NMEA spec says sentences are 80 chars max.

12

This time I've really tried to just really simple, and I'm not understanding the behaviour at all.
I've adjusted the array size for the gpsString.

I can initalise then close the dataFile object just simply:

File dataFile = SD.open("datalog.txt", FILE_WRITE);  
dataFile.close();

and the file is opened and stored on the SD card. With just this, the code will continue to loop and happily print the gpsString to the monitor - though of course, it's not writing to the SD card.

The loop will continue even if trying to write any string with one element (even just dataFile.println("b"); in between the two lines above): it will open the datafile, but it wont write anything to it.

Though it gets worse: If I try to write anything with more than one element (e.g. dataFile.println("ab"); ), even If I define ANY other string in the main loop, the system behaves like it is resetting itself; it just iterates through the start up sequence over and over.

I'll try to rebuild the code starting from code that successfully writes even just a String, and find where things go wrong, but right now, I'm at a loss to even understand what is happening.

As always, your advice is very much appreciated, and thanks for your help so far.

13

A quick update, I'd noticed a few lines of code that I'd included from 'somewhere' that didnt appear to be doing anything - in fact, I dont know where they came from - yes, silly, I know.

Commenting out these lines:

Sd2Card card;
SdVolume volume;
SdFile root;

These appear to be detritus from a previous iteration where I'd attempted using a different SDcard library.

Removing these (in particular, the 'SdFile root;' line) Stopped the constant resetting I refer to in the post prior to this.

At this point though still no information is being written to the file on the card. I still consider this a major win, though the battle continues.

14

I've deconstructed and reconstructed the entire sketch - a few times over.
The basic elements are

  1. GPS read
  2. Tilt read
  3. ADC (mag) read
  4. terminal write
  5. SD write

1-4 are functional, but cannot do 5.

If I do GPS read, Tiltread (not ADC read) then I can do terminal write and SD write.
If I do GPS read, ADC read (not tilt read) then I can do terminal write and SD write.
If I do tilt read, ADC read (but not GPS read) then I can do terminal write and SD write.

The change in behaviour appears to occur after I include the setup code.

The initialistions of the global variables are below

// -------------------------- set up the SD card drive parameters--------------------------------
const int chipSelect = 4;  // indicates open comms. to SD 

// -------------------------- set up the GPS parameters--------------------------------
NeoSWSerial gpsPort(7, 8); ////GPS RX= Ard pin 8, GPS TX=Ard pin 7 
NMEAGPS GPS;

// -------------------------- set up the ADC parameters--------------------------------
uint8_t adc_address = 0x6E; // 0x68 is the default address for all MCP342x devices

MCP342x adc = MCP342x(adc_address);

// Resolution is 12, 14, 16,or 18; gain is 1, 2, 4, or 8.
MCP342x::Config config(MCP342x::channel1, MCP342x::oneShot,MCP342x::resolution16, MCP342x::gain1);
MCP342x::Config status;// Configuration/status read back from the ADC

// -------------------------- set up the tilt / accel. params--------------------------------
uint8_t tilt_address = 0x1D; //address for tilt is at 0x1D, make sure two addresses are different.
MMA_7455 mySensor = MMA_7455(); //Make an instance of MMA_7455

And the setup code is;

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  gpsPort.begin(9600);
  Wire.begin();

//+++++++++++++++++++++++  SET up ADC  ++++++++++++++++++++ 
  
 // Check ADC present
  Wire.requestFrom(adc_address, (uint8_t)1);
  if (!Wire.available()) {
    Serial.print("No adc device found at address ");
    Serial.println(adc_address, HEX);
    while (1);
  } else {Serial.println("Found ADC");}

 ///+++++++++++++++++++++++  SET up TILT sensor  ++++++++++++++++++++ 
  mySensor.initSensitivity(2); // set up  for "Tilt" measurements: 2 = 2g, 4 = 4g, 8 = 8g
    
  // Check TILT present
  Wire.requestFrom(tilt_address, (uint8_t)1);
  if (!Wire.available()) {
    Serial.print("No tilt device found at address ");
    Serial.println(tilt_address, HEX);
    while (1);
  }else {Serial.println("Found TILT");}

///+++++++++++++++++++++++  SET up SD card drive ++++++++++++++++++++ 

  Serial.print("Initializing SD card...");

  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    while (1);
  }else {
    SD.remove("datalog.txt");
    delay(1000); // wait 1 sec for system to settle.
    Serial.println("card initialized.");
    }

   ///+++++++++++++++++++++++  SET up GPS ++++++++++++++++++++ 
  Serial.println("Initialising GPS - please wait.");
  if (GPS.available( gpsPort )){
    gps_fix fix = GPS.read();
    if (fix.valid.location) {Serial.println("..GPS activated and operating");}}
    
  while (!Serial) {;}  // wait for serial port to connect. Needed for native USB port only


}

and the main loop is;

void loop() {
   char gpsString[85];          // container for output string
   char lat[14],lon[14];        // string container for lat, long, type double
   uint8_t dummyRead;           // container for ADC call response
   long magX, magY, magZ;       // container for Mag readings.
  
  if (GPS.available( gpsPort )) {
    gps_fix fix = GPS.read();
    if (fix.valid.location) {

// Get lat lon (double) conv. to string
     dtostrf(fix.latitude(),13, 9, lat);
     dtostrf(fix.longitude(),13, 9, lon);

// Get magnetometer, x, y, z
     dummyRead = adc.convertAndRead(MCP342x::channel1, MCP342x::oneShot,
              MCP342x::resolution14, MCP342x::gain1,10, magX, status);
     dummyRead = adc.convertAndRead(MCP342x::channel3, MCP342x::oneShot,
              MCP342x::resolution14, MCP342x::gain1,10, magY, status);
     dummyRead = adc.convertAndRead(MCP342x::channel4, MCP342x::oneShot,
              MCP342x::resolution14, MCP342x::gain1,10, magZ, status);
              
     
     //format yy mm dd hr:mn:sc:mss ----------lat ----------lon Tix Tiy Tiz --------MX --------MY --------MZ
     sprintf(gpsString, "%2d %2d %2d %2d:%2d:%2d.%03d %13s %13s %5d %5d %5d",
     fix.dateTime.year, fix.dateTime.month, fix.dateTime.date,                          // date
     fix.dateTime.hours, fix.dateTime.minutes, fix.dateTime.seconds, fix.dateTime_ms(), //time
     lat,lon,                                                                           //lat, long
     magX, magY, magZ);                                                                 // Magnetometer.
        
      Serial.println(gpsString);

      File dataFile = SD.open("datalog.txt", FILE_WRITE);
      dataFile.println(gpsString);
      dataFile.close();    

      } // fix.valid.location
    } // GPS.available

}

The main loop does not include any code to read the tiltmeter - just having the setup code in the setup loop is enough to block writing to the file on the SDcard, though it DOES write it to the terminal.

So I'm not sure how useful any of this is - I feel only slightly closer, but just as far away from a solution. At least it's stopped crashing and resetting.

Any thoughts on what might be going on after I run the setup code for tiltmeter/GPS/ADC? A clash of some variables? Any input again, is so incredibly welcome..

Thanks.

15

I believe the problem is solved - though through luck more than anything else.

In the setup, I made calls through I2C to both the ADC (magnetometer) and the tiltmeter. I was really applying these a bit blindly and I noticed that it was when both were present in the setup, that the data were not written to the SDcard.

Simply removing the explicit setup to the ADC seemed to fix it - though I am not entirely clear the magical mechanism that is being used to receive data from the ADC, nor do I really understand why any of that might have an impact on data being written into the file on the SD card.

For now, the problem seems solved - though I havent really tested the integrity/validity of the data from either the ADC or tiltmeter very robustly.

I'm attaching the sketch that I have for now, with the offending section commented out.

sketch_may27d.ino (5.49 KB)