Failure to write an Array to an SD card file

As suggested to me in a prior Topic I had posted ( previous Topic) , I am re-working my program to write data pairs (time, force) to an Array first, then attempting to loop through that array and write all the data-pairs to an SD card. Here is the background of my setup:

  • Generic {SD-card + RTC} shield, using the DS1307Z chip
  • Arduino UNO R4 Minima
  • Arduino 2.8.3 IDE (running on a Linux platform)

Note that the {SD-card + RTC} Shield I am now using is NOT the same as used in the prior Topic that I noted above; that hardware is actually in use and I did not have a duplicate of the Adafruit shield. As a result, there are multiple changes to the code.

I have proved that the hardware works, by using Arduino Examples for the DS1307Z and the SD-card.

The re-write is partially successful. I added many comments to figure out where the execution stopped. The sticking point, now, is where I try to loop-through the Array, and write data-pairs to the SD card. After several hours of focus on this, I still cannot see what the problem is. There are no compilation-time errors. A fresh set of eyes and some insight born of experience would be greatly appreciated. Here is my code; I have added a comment ("<<<<< output terminates abruptly here!") where the execution appears to hang up / terminate.


#include <SD.h>
#include "HX711.h"
//#include <Wire.h"
#include <I2C_RTC.h>

static DS1307 RTC;

#define SD_FAT_TYPE 3
#define PIN_SPI_CS 4
#define FILE_NAME "test.txt"
uint8_t dataPin = 2;
uint8_t clockPin = 3;  //3
const int chipSelect = 10;
File myFile;
HX711 scale;

float f;
float Unixtime;
int i = 0;  //iteration counter
int j = 0;  //iteration counter
int k = 0;  //iteration counter
int n = 20;  // number of 1-second interval iterations

float matrix[20][1];  // for up to 21 data pairs of [unixtime], [force]



void setup() {

  scale.begin(dataPin, clockPin);  // setup the HX711 module
  scale.set_scale(27500);          // scale output from the 1 kg beam so that output is in Gram units
  scale.tare();                    // tare the scale output


  Serial.begin(115200);  // Open serial communications, and wait for a port to open:
  while (!Serial)        // wait for Serial Monitor to connect. Needed for native USB port boards only:
    ;
  Serial.print("Initializing SD card...");

  if (!SD.begin(chipSelect)) {
    Serial.println("initialization failed. Things to check:");
    Serial.println("1. is a card inserted?");
    Serial.println("2. is your wiring correct?");
    Serial.println("3. did you change the chipSelect pin to match your shield or module?");
    Serial.println("Note: press reset button on the board and reopen this Serial Monitor after fixing your issue!");
    while (true)
      ;
  }
  Serial.println("initialization done.");

  if (RTC.isConnected() == false) {
    Serial.println("RTC Not Connected!");
    while (true)
      ;
  }
  Serial.println("*** RTC 1307 ***");
  Serial.print("Is Clock Running : ");
  if (RTC.isRunning())
    Serial.println("Yes!");

  myFile = SD.open("test.txt", FILE_WRITE);  // open the SD file. Note that only one file can be open at a time, so you have to close this file before opening another.

  if (myFile) {  // if the file opened okay, write to it:
    Serial.print("Writing to test.txt ...");
    myFile.println("testing 999.");

    myFile.close();  // close the file:
    Serial.println("SD File now closed.");

  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt ");
  }


  Serial.println("Reading & display file contents");
  myFile = SD.open("test.txt");  // re-open the file for reading contents
  if (myFile) {
    Serial.println("test.txt :");
    while (myFile.available()) {  // read data from the file until the end:
      Serial.write(myFile.read());
    }
    myFile.close();  // close the file:
  } else {           // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void loop() {
  while (i < n) {  //time-and-weight measuring steps at 1 second intervals
    Serial.print(RTC.getDateTimeString());
    Serial.println("");
    Unixtime = RTC.getEpoch();
    // f = scale.get_units(5);
    matrix[i][0] = Unixtime;
    matrix[i][1] = scale.get_units(5);
    Serial.println(matrix[i][0]);
    Serial.println(matrix[i][1]);

    delay(1000);
    i = i + 1;
  }

  myFile = SD.open("test.txt", FILE_WRITE);  // open SD file to write the data stored in 'matrix'
  if (myFile) {
    Serial.print("Now Writing 'matrix' to test.txt ...");
    myFile.println(",,");  //blanks as a data-block delimiter

    while (j < n) {
      Serial.println("first step");  ///   <<<<< output terminates abruptly here!>>>>>>>>>>>>>
      myFile.print(matrix[j][0]);
      myFile.print(matrix[j][1]);
      Serial.println("second step");
      j = j + 1;
    }
  } else {
    Serial.println("error opening test.txt ");  // if the file didn't open, print an error
    myFile.close();

    // Serial.print(RTC.getDateTimeString());
    // Serial.print(", Unix Time: ");
    // Serial.println(RTC.getEpoch());
    // f = scale.get_units(5);
    // Serial.println(f);
    // Serial.print("Done iterations and file closed");
  }
}

Where in your code to you reset j to 0 for the next time through the loop?

The same question applies to i in the previous while loop.

matrix[i][0] = Unixtime;
matrix[i][1] = scale.get_units(5);

I think you are writing outside the bounds of your declared 2D array as you have only declared there will be one value of the second index. You have the index numbers correct for zero based array indexes but you need to your matrix declared as [20][2]. The index values are 0 through 19 for the first value with 0 and 1 as the second.

I might not understand your code but are you sure that that is a two-dimensional array? The second dimension ([1]) indicates that it's actually a one-dimensional array.
And where does the 21 come from?

You might want to consider using a struct (or array of structs).

It's not advisable to use single character global variables; if you have to search your code to find where the variable i is used you will find 279 characters. I don't think that they all refer to the variable i :wink:

Lastly, you never close the file and every iteration through loop() you open the file which was not closed; I did not check how the SD library handles that but it is bad coding practice in my opinion.
But the bigger problem is that when you eject the card it might only have buffered and not have written the latest data to the file.

Perhaps this gets the cause and effect backward: if the declaration was for the max index, then you'd get 21 and 2. But it's the declaration is for the number of elements, which are base-zero. So the valid indices are [0 to 19] and [just 0], a single element. Also

If that's for that array of "pairs", a 32-bit float only uses 23 bits for the mantissa. For example, noon 12-May GMT is 1'778'569'200 seconds since 1970. The closest you can get is 1'778'569'216. The time before that (closest before noon) is 1'778'569'088 -- over a two minute gap.

Use a struct to solve both -- or at least one-and-a-half -- of the problems; using the proper type for the time. time_t is an opaque type, but almost always a big integer for the number of seconds.

struct {
  time_t Unixtime;
  float force;
} pairs[21];

// ...

  pairs[i].Unixtime = RTC.getEpoch();
  pairs[i].force = scale.get_units(5);

You only close the file if you haven't succeeded in opening it.

On another note, If you know how many bytes the data is you are writing and you have a pointer to those bytes, you may as well just use SD.write() instead of print()

Point taken, regarding single-character variables.

My earliest coding experience was with punch cards. We didn't have the luxury of "search and replace" functionality. What we did have was: "Oops, I dropped my entire box of cards on the staircase, now I am f_____d". That was more serious than finding too many instances of "i" !!!

Thanks for catching that.

I have moved myFile.close() to sit outside ( = below) the " }"

Thanks....point taken, and it could very well be that 'struct' is a better option. But (for now) I am going to continue using the Array until I am more comfortable with that construction and learn more about its limitation in this application. ...

I have used your suggestion to changing the Type of Unixtime to time_t .

I had hoped that using time_t might also solve another problem with my program which I had not noted previously in this Topic ----- my "Unixtime" value is not updating, even though it is being recorded on the SD card now with 2 decimals of precision.

Note that RTC.getDateTimeString() is updating and reporting the current time with each iteration. But RTC.getEpoch() seems to fetch an initial value, and then it repeats the same value with each iteration. This seems to be the most pressing issue I have at this moment

Here is my current code. It is now working (mostly) ⸺ it iterates, reports the values to the serial monitor correctly, apparently updating the Array correctly, and then it iteratively writes all the Array values to the SD card as comma-separated, ordered pairs:

#include <SD.h>
#include "HX711.h"
//#include <Wire.h"
#include <I2C_RTC.h>

static DS1307 RTC;

#define SD_FAT_TYPE 3
#define PIN_SPI_CS 4
#define FILE_NAME "test.txt"
uint8_t dataPin = 2;
uint8_t clockPin = 3;  //3
const int chipSelect = 10;
File myFile;
HX711 scale;

float f;
time_t Unixtime;
int i = 0;  //iteration counter
int j = 0;  //iteration counter
int k = 0;  //iteration counter
int n = 5;  // number of 1-second interval iterations

float matrix[20][2];  // for up to 21 data pairs of [unixtime], [force]



void setup() {

  scale.begin(dataPin, clockPin);  // setup the HX711 module
  scale.set_scale(27500);          // scale output from the 1 kg beam so that output is in Gram units
  scale.tare();                    // tare the scale output


  Serial.begin(115200);  // Open serial communications, and wait for a port to open:
  while (!Serial)        // wait for Serial Monitor to connect. Needed for native USB port boards only:
    ;
  Serial.print("Initializing SD card...");

  if (!SD.begin(chipSelect)) {
    Serial.println("initialization failed. Things to check:");
    Serial.println("1. is a card inserted?");
    Serial.println("2. is your wiring correct?");
    Serial.println("3. did you change the chipSelect pin to match your shield or module?");
    Serial.println("Note: press reset button on the board and reopen this Serial Monitor after fixing your issue!");
    while (true)
      ;
  }
  Serial.println("initialization done.");

  if (RTC.isConnected() == false) {
    Serial.println("RTC Not Connected!");
    while (true)
      ;
  }
  Serial.println("*** RTC 1307 ***");
  Serial.print("Is Clock Running : ");
  if (RTC.isRunning())
    Serial.println("Yes!");

  myFile = SD.open("test.txt", FILE_WRITE);  // open the SD file. Note that only one file can be open at a time, so you have to close this file before opening another.

  if (myFile) {  // if the file opened okay, write to it:
    Serial.print("Writing to test.txt ...");
    myFile.println("testing 999.");

    myFile.close();  // close the file:
    Serial.println("SD File now closed.");

  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt ");
  }


  Serial.println("Reading & display file contents");
  myFile = SD.open("test.txt");  // re-open the file for reading contents
  if (myFile) {
    Serial.println("test.txt :");
    while (myFile.available()) {  // read data from the file until the end:
      Serial.write(myFile.read());
    }
    myFile.close();  // close the file:
  } else {           // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void loop() {
  j = 0;
  i = 0;
  while (k < 1) {

    while (i < n) {  //time-and-weight measuring steps at 1 second intervals
      Serial.print(RTC.getDateTimeString());
      Serial.println("");
      Unixtime = RTC.getEpoch();
      // f = scale.get_units(5);
      matrix[i][0] = Unixtime;
      matrix[i][1] = scale.get_units(5);
      Serial.println(matrix[i][0]);
      Serial.println(matrix[i][1]);
      delay(1000);
      i = i + 1;
    }

    myFile = SD.open("test.txt", FILE_WRITE);  // open SD file to write the data stored in 'matrix'
    if (myFile) {
      Serial.println("Now Writing 'matrix' to test.txt ...");

      while (j < n) {
        Serial.print(j);
        Serial.println("first step");  ///   <<<<< output terminates abruptly here!>>>>>>>>>>>>>
        myFile.print(matrix[j][0]);
        myFile.print(",");
        myFile.print(matrix[j][1]);
        myFile.print(",");
        myFile.println("");
        Serial.print(j);
        Serial.println("second step");
        j = j + 1;
      }
      myFile.println(",,,");  //blanks as a data-block delimiter

    } else {
      Serial.println("error opening test.txt ");  // if the file didn't open, print an error
    }
    
    myFile.close();
    break;

    // Serial.print(RTC.getDateTimeString());
    // Serial.print(", Unix Time: ");
    // Serial.println(RTC.getEpoch());
    // f = scale.get_units(5);
    // Serial.println(f);
    // Serial.print("Done iterations and file closed");
  }
  k = k + 1;
}

Here is an example of the Output ⸺ note that the RTC.getDateTimeString() actually increments as expected, but that RTC.getEpoch() does not increment (i.e., within the loop..but that Epoch does increment on successive loop-to-loop runs of the program). I have added emphasis and notes, below, to draw attention to this fact.

Initializing SD card...initialization done.
*** RTC 1307 ***
Is Clock Running : Yes!
Writing to test.txt ...SD File now closed.
Reading & display file contents
test.txt :
testing 999.
1778602752.00,-0.00,
1778602752.00,-0.00,
1778602752.00,-0.00,
1778602752.00,-0.00,
1778602752.00,-0.00,
,,,
testing 999.
MON 12-05-2026 16:18:44
1778602752.00
-0.00
MON 12-05-2026 16:18:46 note that the seconds increments
1778602752.00 note that the Epoch does not increment
-0.00
MON 12-05-2026 16:18:47
1778602752.00
-0.00
MON 12-05-2026 16:18:48
1778602752.00
-0.00
MON 12-05-2026 16:18:50
1778602752.00
-0.00
Now Writing 'matrix' to test.txt ...
0first step
0second step
1first step
1second step
2first step
2second step
3first step
3second step
4first step
4second step

It's lying to you. It is the exact same problem I mentioned earlier. There aren't 2 decimals of precision. At this point, each available value is 128 apart: over two minutes, since it is counting seconds. In effect, worse than "negative 2 decimals" of precision.

See for yourself

  1. Right-click this link to the not-incrementing Epoch, 1778602752.00 to open it in another tab. That page deconstructs the float value
  2. Note that it drops a decimal place: it's not going to lie to you as much: 1778602752.0
  3. A 32-bit float comprises three parts. The range of bits are underlined with different colors. In this case
    • 1-bit Sign (red): 0 means it's positive
    • 8-bit Exponent (green). It's biased so that it's always a non-negative value. 255 is the max value; 128 is in/near the middle. So the value 157 is about 2 to the 30th power for the magnitude, which sounds right.
    • 23-bit Significand (aka mantissa, blue): all the remaining bits to encode the actual value of that magnitude

But wait, if the magnitude of the value is 230, that means it requires 30 bits. There are only 23. That means the last seven bits are just missing, and each successive value increments by 27, 128.

  1. Note the value of the significand: 5506726 (this is shown as decimal). The first bit underline blue is 1. You can do the math to confirm this value is between 222 and 223
  2. Click the + control to increment the significand to 5506727
  3. The value is now 1778602880.0 -- 128 more than before

Worst case, you have to wait over two minutes for the float value to increment; on average, about a minute.

Another way to look at it

void setup() {
  Serial.begin(115200);
}

float start = 1778602752.0; 

void loop() {
  static uint32_t i = start;
  ++i;
  float f = i;
  Serial.print(i);
  Serial.print('\t');
  Serial.print(i, BIN);
  Serial.print('\t');
  Serial.println(f, 1);
  while (f != start) delay(10);
}

The end of that

1778602812	1101010000000110101001100111100	1778602752.0
1778602813	1101010000000110101001100111101	1778602752.0
1778602814	1101010000000110101001100111110	1778602752.0
1778602815	1101010000000110101001100111111	1778602752.0
1778602816	1101010000000110101001101000000	1778602752.0
1778602817	1101010000000110101001101000001	1778602880.0

That last integer rounds up to the next representable float.

Ohhhhhhhhhhh.

Thanks for taking the considerable time (and effort!) to provide a very clear insight. The "penny did not drop" the first time around.

So, in short: if I want / need seconds-level resolution I must abandon "Epoch" and go back to the RTC.getDateTimeString() and the related queries of the RTC system. Unfortunate, as Epoch seemed like a much more elegant solution....

Since you're on Uno R4, you could use double, which is a 64-bit floating point number with a 52-bit significand. While you're at it, use double for the force as well: a matrix of double.

But a struct would be better.

Kenb4-- Thanks for the suggestion of double-float. I have never used double in Arduino before, so thank you for pointing out that it is possible to do so with the Uno R4.

The use of double definitely solves the problem of using Epoch dates (down to the seconds-level). Having got this far, I am going to spend some time 'cleaning up' my existing code, and then I will attempt to make use of struct.

Today, I did notice some peculiar activity with my Uno R4, while USB-connected to my Linux-box and with the Arduino IDE in "serial monitoring" mode: if I left my computer for long enough that the security lock-screen activates, it seems as though logging back into the Linux causes my code (as it is now written) to re-activate in an endless loop mode. Normally, the loop is written to terminate after a once-through pass, such that successive data-collection loops requires the use of the on-board RESET button (not elegant, but it works for now)....here is the code as it exists today:

#include <SD.h>
#include "HX711.h"
//#include <Wire.h"
#include <I2C_RTC.h>

static DS1307 RTC;

#define SD_FAT_TYPE 3
#define PIN_SPI_CS 4
#define FILE_NAME "test.txt"
uint8_t dataPin = 2;
uint8_t clockPin = 3;  //3
const int chipSelect = 10;
File myFile;
HX711 scale;

double f;
double Unixtime;
int i = 0;  //iteration counter
int j = 0;  //iteration counter
int k = 0;  //iteration counter
int n = 20;  // number of 1-second interval iterations

double matrix[20][2];  // for up to 21 data pairs of [unixtime], [force]



void setup() {

  scale.begin(dataPin, clockPin);  // setup the HX711 module
  scale.set_scale(2750);          // 
  scale.tare();                    // tare the scale output


  Serial.begin(115200);  // Open serial communications, and wait for a port to open:
  while (!Serial)        // wait for Serial Monitor to connect. Needed for native USB port boards only:
    ;
  Serial.print("Initializing SD card...");

  if (!SD.begin(chipSelect)) {
    Serial.println("initialization failed. Things to check:");
    Serial.println("1. is a card inserted?");
    Serial.println("2. is your wiring correct?");
    Serial.println("3. did you change the chipSelect pin to match your shield or module?");
    Serial.println("Note: press reset button on the board and reopen this Serial Monitor after fixing your issue!");
    while (true)
      ;
  }
  Serial.println("initialization done.");

  if (RTC.isConnected() == false) {
    Serial.println("RTC Not Connected!");
    while (true)
      ;
  }
  Serial.println("*** RTC 1307 ***");
  Serial.print("Is Clock Running : ");
  if (RTC.isRunning())
    Serial.println("Yes!");

  myFile = SD.open("test.txt", FILE_WRITE);  // open the SD file. Note that only one file can be open at a time, so you have to close this file before opening another.

  if (myFile) {  // if the file opened okay, write to it:
    Serial.print("Writing to test.txt ...");
    myFile.println("testing 999.");

    myFile.close();  // close the file:
    Serial.println("SD File now closed.");

  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt ");
  }


  Serial.println("Reading & display file contents");
  myFile = SD.open("test.txt");  // re-open the file for reading contents
  if (myFile) {
    Serial.println("test.txt :");
    while (myFile.available()) {  // read data from the file until the end:
      Serial.write(myFile.read());
    }
    myFile.close();  // close the file:
  } else {           // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void loop() {
  j = 0;
  i = 0;
  while (k < 1) {

    while (i < n) {  //time-and-weight measuring steps at 1 second intervals
      Serial.print(RTC.getDateTimeString());
      Serial.println("");
      Unixtime = RTC.getEpoch();
      // f = scale.get_units(5);
      matrix[i][0] = Unixtime;
      matrix[i][1] = scale.get_units(5);
      Serial.println(matrix[i][0]);
      Serial.println(matrix[i][1]);
      delay(1000);
      i = i + 1;
    }

    myFile = SD.open("test.txt", FILE_WRITE);  // open SD file to write the data stored in 'matrix'
    if (myFile) {
      Serial.println("Now Writing 'matrix' to test.txt ...");

      while (j < n) {
        Serial.print(j);
        Serial.println("first step");  ///   <<<<< output terminates abruptly here!>>>>>>>>>>>>>
        myFile.print(matrix[j][0]);
        myFile.print(",");
        myFile.print(matrix[j][1]);
        myFile.print(",");
        myFile.println("");
        Serial.print(j);
        Serial.println("second step");
        j = j + 1;
      }
      myFile.println(",,,");  //blanks as a data-block delimiter

    } else {
      Serial.println("error opening test.txt ");  // if the file didn't open, print an error
    }
    
    myFile.close();
    Serial.println("SD card file is closed");
    break;

    // Serial.print(RTC.getDateTimeString());
    // Serial.print(", Unix Time: ");
    // Serial.println(RTC.getEpoch());
    // f = scale.get_units(5);
    // Serial.println(f);
    // Serial.print("Done iterations and file closed");
  }
  k = k + 1;
}

This is another timing issue. It's due to the way you're controlling execution. Stripped down, the sketch is essentially

int k = 0;

void setup() {
  Serial.begin(115200);
}

void loop() {
  while (k < 1) {
    Serial.println("do something");
    break;
  }
  ++k;
}

Note that int is signed. I added some extra code to estimate how long until it re-activates, which itself affects the timing somewhat

Spoiler
void loop() {
  if ((k & 0xFFFFF) == 0xFFFFF) {
    auto now = millis();
    Serial.print(k, HEX);
    Serial.print(" countdown: ");
    Serial.println((1LL << 31) * now / k - now);
  }

  while (k < 1) {
    if (k) {
      Serial.print("it's negative; wrap-around at: ");
      Serial.println(millis());
      for (;;);
    } else {
      Serial.println("it's zero");
    }

    break;
  }
  ++k;
}

Just under a half-hour

A run once situation is well handled with a boolean control variable rather than a numberical index.

boolean runOnce = true; 

void setup() {
  Serial.begin(115200);
  while(!Serial);
}

void loop() {
  if(runOnce) {
    Serial.println("do something");
    runOnce =  false;
  }

}

I incorporated your suggestion. I am sure it will work as you said but I am running a test (greater than ~ 1/2 hour, as suggested by the calculation of Kenb4), just to prove it to myself....!

Fascinating.....so the "break" actually does not terminate the execution of the loop (but in effect it is more like a "brake" ?). Meanwhile, the incrementing ++k runs in the background until it fills the buffer space and then it effectively counts downward??

break does terminate the while. But the loop function is executed repeatedly (as shown in my demo program at the end of post #10). As may know, C/C++ programs have a main function. For an .ino sketch, that is essentially

void main() {
  setup();   // you provide this
  for (;;) {
    loop();  // and this
  }
}

As for incrementing a signed value, try a type with a smaller range

void setup() {
  Serial.begin(115200);
  delay(750);
}

signed char x = 0;

void loop() {
  Serial.println(x);
  if (!++x) {
    Serial.println("done");
    for (;;);
  }
}

Takes about a second.