Did I just discover "overflow"? All of a sudden, Arduino loop code cannot open the text file (but setup can)

Barometric Pressure Sensor to SD card on Arduino Uno.

Update 2:
Updated debugging code

Update:
void setup works fine.
Despite working fine and recording data from my BMP sensor, void loop no longer likes my character arrays (or the sensor), and I was able to get the text file to be written to if I comment out all the arrays and sensor stuff.

The moment I add a single character array and a sprintf() with one array in it, I get the "cannot open Zorro.txt" error again.

Original post:

I built this code up piecemeal, adding each array one at a time, integrating each array into sprintf(), and everything was working just fine until I added the last two header titles.

The SD card works (Arduino's SD examples "Datalogger" works, and "Read/Write" works... meaning I can open a test.txt file and write Testing 1,2,3 to it).

I've tried:

  1. commenting out arrays and removing them from sprintf
  2. replacing header strings with simple text
  3. increasing hypstr buffer size to 256
  4. changing text file names, moving original text file into another folder (short of erasing the SD card, I don't feel this is necessary)
  5. disconnecting sensor, running code (proper error message works) ,reconnecting sensor

Up to the last moment, I was able to write milliseconds + seven arrays of data... and then everything broke:

145596,25.90,78.62,18841,1517.53,4978.46,1570.41,5152.56
147802,25.90,78.62,18836,1518.11,4980.36,1571.18,5154.47
150006,25.90,78.62,18839,1517.82,4980.04,1570.79,5153.21
152211,25.90,78.62,18841,1517.43,4976.87,1570.60,5153.21

Any advice would be fantastic, I'm a noob, so I really can't tell what went wrong. Hardware/wires seem to work (replaced the cs = 4 wire just in case, but again, simple codes do write to the SD card).

/* C
SD Card Template to use for testing
REMEMBER: Pin 10 must be set as an output and cannot be used for anything!!! (SD card code stuff)

SD card setup
** MOSI (DI) - pin 11
 ** MISO (DO) - pin 12
 ** CLK - pin 13
 ** CS - pin 4 (for MKRZero SD: SDCARD_SS_PIN)
*/
//Libraries
#include <SPI.h>
#include <SD.h>
#include <Adafruit_BMP085.h>

//BMP Definitions
int press;  //int value is from -32768 to 32767
Adafruit_BMP085 bmp;
uint32_t timeStamp = 0;

//LED <---note to arduino forum, not used just yet
const int ledPin = 9;
// DO NOT USE PIN 10

//Document Info
String header1 = ("Begin BMP Testing");  //update as neededβ€”test done
String header2 = ("Time,TempC,TempF,Pressure(Pa),Alt(m,1013.25mb),Alt(ft,1013.25mb),rAlt(m,1019.9mb),rAlt(ft,1019.9mb)");

//SD Card
File dataFile;     //name of file object
const int cS = 4;  //SD card write

void setup() {
  Serial.begin(115200);
  while (!Serial) {
    ;  // wait for serial port to connect. Needed for native USB port only
  }

  pinMode(ledPin, OUTPUT);
  pinMode(10, OUTPUT);  //Required for SD card module, do not use

  //SD CARD
  Serial.print("Initializing SD card...");
  // see if the card is present and can be initialized:
  if (!SD.begin(cS)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    while (1)
      ;
  }
  Serial.println("card initialized.");

  //Print initial startup
  dataFile = SD.open("zorro.txt", FILE_WRITE);
  if (dataFile) {
    dataFile.println("");
    dataFile.println(header1);  //headers
    dataFile.println(header2);
    dataFile.close();
    // debug
    Serial.println("");
    Serial.println(header1);
    Serial.println(header2);
  }
  //Start BMP
  if (!bmp.begin()) {
    Serial.println("Could not find a valid BMP085 sensor, check wiring!");
    while (1) {}
  }
}

void loop() {
  timeStamp = millis();  // this declaration must be in void loop

  // SD CARD
  dataFile = SD.open("zorro.txt", FILE_WRITE);
  if (dataFile) {
    dataFile.print(timeStamp);
    dataFile.print(",");
    //debug
    Serial.print(timeStamp);
    Serial.print(",");
  //BMP Sensor
  //tempC
    char tempCStr[7];
    float tempC = bmp.readTemperature();
    dtostrf(tempC, 1, 2, tempCStr);
  //tempF
    char tempFStr[7];
    float tempF = bmp.readTemperature() * (9.0 / 5.0) + 32;
    dtostrf(tempF, 1, 2, tempFStr);
  //pressure
    press = bmp.readPressure();
  //alt (m) – 101325 Pa
    char altMStr[12];
    float altM = bmp.readAltitude();
    dtostrf(altM, 1, 2, altMStr);
  //alt (ft) β€” 101325 Pa
    char altFtStr[12];
    float altFt = bmp.readAltitude() * 3.28084;
    dtostrf(altFt, 1, 2, altFtStr);
  //real alt (m) – 101500 Pa
    char raltMStr[12];
    float raltM = bmp.readAltitude(101990);
    dtostrf(raltM, 1, 2, raltMStr);
  //real alt (ft) β€” 101300 Pa
    char raltFtStr[12];
    float raltFt = bmp.readAltitude(101990) * 3.28084;
    dtostrf(raltFt, 1, 2, raltFtStr);

    char hypstr[256];
    sprintf(hypstr, "%s,%s,%d,%s,%s,%s,%s", tempCStr, tempFStr, press, altMStr, altFtStr, raltMStr, raltFtStr);

    dataFile.println(hypstr);
    Serial.println(hypstr);

    dataFile.close();

  } else {
    Serial.println("error opening zorro.txt");
  }
  //END SD card write

  delay(2000);
}

Please to be more clear about "everything broke". What does it means regarding to the way how the code works.

1 Like

You are opening the same file in setup() and loop(). Try closing the file at the end of setup()).

1 Like

But I do close it in setup: (note: I changed object name to myFile in one of my attempts at trying to fix things)

 myFile = SD.open("zorro.txt", FILE_WRITE);
  if (myFile) {
    myFile.println("");
    myFile.println(header1);  //headers
    myFile.println(header2);
    myFile.close();

Everything was working swell until I reset the Arduino and suddenly I get "cannot open Zorro.txt"...

I've been debugging step by step. The void setup works fine. I'll leave it as is.

Void loop suddenly works if I comment out all of the character arrays and sprintf function. But it was working just fine earlier! I'm so confused if maybe the sensor itself is messing things up? Anyway, I'll try adding in one simple array and trying to print it to the text file as my next step. (Second attempt)

void loop() {
  timeStamp = millis();  // this declaration must be in void loop

Serial.println("Step 1"); //works to this point

  // SD CARD
  myFile = SD.open("zorro.txt", FILE_WRITE);
  if (myFile) {
    myFile.print(timeStamp);
    myFile.print(",");
    //debug 
    Serial.print(timeStamp);
    Serial.print(",");
/*  //BMP Sensor
  //tempC
    char tempCStr[7];
    float tempC = bmp.readTemperature();
    dtostrf(tempC, 1, 2, tempCStr);
  //tempF
    char tempFStr[7];
    float tempF = bmp.readTemperature() * (9.0 / 5.0) + 32;
    dtostrf(tempF, 1, 2, tempFStr);
 /* //pressure
    press = bmp.readPressure();
  //alt (m) – 101325 Pa
    char altMStr[12];
    float altM = bmp.readAltitude();
    dtostrf(altM, 1, 2, altMStr);
  //alt (ft) β€” 101325 Pa
    char altFtStr[12];
    float altFt = bmp.readAltitude() * 3.28084;
    dtostrf(altFt, 1, 2, altFtStr);
  //real alt (m) – 101500 Pa
    char raltMStr[12];
    float raltM = bmp.readAltitude(101990);
    dtostrf(raltM, 1, 2, raltMStr);
  //real alt (ft) β€” 101300 Pa
    char raltFtStr[12];
    float raltFt = bmp.readAltitude(101990) * 3.28084;
    dtostrf(raltFt, 1, 2, raltFtStr);

    char hypstr[256];
    sprintf(hypstr, "%s,%s,%d,%s,%s,%s,%s", tempCStr, tempFStr, press, altMStr, altFtStr, raltMStr, raltFtStr);

    char hypstr[256]; //testing min arrays
    sprintf(hypstr, "%s,%s",tempCStr,tempFStr); //testing min arrays

    myFile.println(hypstr); 
    Serial.println(hypstr);
*/
    myFile.close();

  } else {
    Serial.println("error opening zorro.txt");
  }
  //END SD card write

  delay(2000);
}

What is your arduino board?

Uno. Apologies for omitting that!

I'll post an update in the original post. I feel like I can get things to work so long as I don't 'use the sensor :joy:

Why did you created a new array for each sprintf conversion? Create a one with maximum required size (in your case - 256 bytes) and reuse it for all other followed conversions.

oh that looks like me debugging stuff (I was moving /* and */ comment-out symbols up and down the code... I forgot to delete the second sprintf thing. If I comment out alllll the arrays, and start over with one single array (say, TempC) and one sprintf with one array, I get the "cannot open text file" error.

Which is so weird because it was literally working for every array that I added, one by one, going very slowly.

I'll do another test and reply to you my void loop code

This works, and will write
"debug 1"
"[time], debug 2"

to the text file:

void loop() {
  timeStamp = millis();  // this declaration must be in void loop

Serial.println("debug 1"); //works to this point

  // SD CARD
  myFile = SD.open("zorro.txt", FILE_WRITE);
  if (myFile) {
    myFile.print(timeStamp);
    myFile.print(",");
    //debug 
    Serial.print(timeStamp);
    Serial.print(",");
 /* //BMP Sensor
  //tempC
    char tempCStr[7];
    float tempC = bmp.readTemperature();
    dtostrf(tempC, 1, 2, tempCStr);
/*  //tempF
    char tempFStr[7];
    float tempF = bmp.readTemperature() * (9.0 / 5.0) + 32;
    dtostrf(tempF, 1, 2, tempFStr);
 /* //pressure
    press = bmp.readPressure();
  //alt (m) – 101325 Pa
    char altMStr[12];
    float altM = bmp.readAltitude();
    dtostrf(altM, 1, 2, altMStr);
  //alt (ft) β€” 101325 Pa
    char altFtStr[12];
    float altFt = bmp.readAltitude() * 3.28084;
    dtostrf(altFt, 1, 2, altFtStr);
  //real alt (m) – 101500 Pa
    char raltMStr[12];
    float raltM = bmp.readAltitude(101990);
    dtostrf(raltM, 1, 2, raltMStr);
  //real alt (ft) β€” 101300 Pa
    char raltFtStr[12];
    float raltFt = bmp.readAltitude(101990) * 3.28084;
    dtostrf(raltFt, 1, 2, raltFtStr);

    char hypstr[256];
    sprintf(hypstr, "%s,%s,%d,%s,%s,%s,%s", tempCStr, tempFStr, press, altMStr, altFtStr, raltMStr, raltFtStr);

    myFile.println(hypstr); 
    Serial.println(hypstr);
*/
    myFile.println("debug 2"); 
    Serial.println("debug 2");
    
//    char hypstr[256]; //testing min arrays
//    sprintf(hypstr,"%s",tempCStr); //testing min arrays

//    myFile.println(hypstr); 
//    Serial.println(hypstr);
    
        myFile.close();

  } else {
    Serial.println("error opening zorro.txt");
  }
  //END SD card write

  delay(2000);
}

Uncommenting one single array, using sprintf for that one array, and suddenly everything stops working. But it WAS working earlier.

void loop() {
  timeStamp = millis();  // this declaration must be in void loop

Serial.println("debug 1"); //works to this point

  // SD CARD
  myFile = SD.open("zorro.txt", FILE_WRITE);
  if (myFile) {
    myFile.print(timeStamp);
    myFile.print(",");
    //debug 
    Serial.print(timeStamp);
    Serial.print(",");
  //BMP Sensor
  //tempC
    char tempCStr[7];
    float tempC = bmp.readTemperature();
    dtostrf(tempC, 1, 2, tempCStr);
/*  //tempF
    char tempFStr[7];
    float tempF = bmp.readTemperature() * (9.0 / 5.0) + 32;
    dtostrf(tempF, 1, 2, tempFStr);
 /* //pressure
    press = bmp.readPressure();
  //alt (m) – 101325 Pa
    char altMStr[12];
    float altM = bmp.readAltitude();
    dtostrf(altM, 1, 2, altMStr);
  //alt (ft) β€” 101325 Pa
    char altFtStr[12];
    float altFt = bmp.readAltitude() * 3.28084;
    dtostrf(altFt, 1, 2, altFtStr);
  //real alt (m) – 101500 Pa
    char raltMStr[12];
    float raltM = bmp.readAltitude(101990);
    dtostrf(raltM, 1, 2, raltMStr);
  //real alt (ft) β€” 101300 Pa
    char raltFtStr[12];
    float raltFt = bmp.readAltitude(101990) * 3.28084;
    dtostrf(raltFt, 1, 2, raltFtStr);

    char hypstr[256];
    sprintf(hypstr, "%s,%s,%d,%s,%s,%s,%s", tempCStr, tempFStr, press, altMStr, altFtStr, raltMStr, raltFtStr);

    myFile.println(hypstr); 
    Serial.println(hypstr);
*/
    
    char hypstr[256]; //testing one array
    sprintf(hypstr,"%s",tempCStr); //testing one array

    myFile.println(hypstr); 
    Serial.println(hypstr);
    
        myFile.close();

  } else {
    Serial.println("error opening zorro.txt");
  }
  //END SD card write

  delay(2000);
}

How much ram is used indicated by the message at the end of the compile? Remember that SD support will add at least 512 bytes to that. Are you running out of ram?

1 Like

That's a great question. This worries me, then:

WAIT hold on, this is after compiling my code which is still being debugged.... let me get back to you in a moment
----"Sketch uses 17804 bytes (55%) of program storage space. Maximum is 32256 bytes.
Global variables use 1409 bytes (68%) of dynamic memory, leaving 639 bytes for local variables. Maximum is 2048 bytes."----- ignore

Here is the compiled info for my full code:

"Sketch uses 21962 bytes (68%) of program storage space. Maximum is 32256 bytes.
Global variables use 1431 bytes (69%) of dynamic memory, leaving 617 bytes for local variables. Maximum is 2048 bytes."

The thing is, I'm supposed to get like 7 or 8 sensors onto this thing and have wondered how on earth I'm supposed to have enough room for all that. Yikes.

  // somewhere above setup()
  void printCommaSpace()
  {
    myFile.print(F( ", ")); 
    Serial.print(F(", "));
  }

    // what you print goes to the 64 char serial output buffer
    // if you overfill that buffer, code execution has to wait until 
    // the buffer is no longer full. 
    // Printing too much quickly is a not a good practice.
 
    myFile.print(tempCStr); 
    Serial.print(tempCStr);
    printCommaSpace();

   myFile.println( tempFStr); 
    Serial.println( tempFStr);
    printCommaSpace();

    myFile.print(press); 
    Serial.print(press);
    printCommaSpace()
   // etc for each
  // after or for the last field printed, println() and write(10) to the file to end the line

The F() macro used in print keeps the text in flash. otherwise it loads to RAM.

The file prints go to the 512 byte SD buffer in RAM. It won't overflow.

I don't know what board you have. On an Uno, saving RAM gets critical pretty quickly.

1 Like

Global variables use 1409 bytes (68%) of dynamic memory, leaving 639 bytes for local variables. Maximum is 2048 bytes

That 639 bytes is all the stack you get as well as all those text headers.

You need a lot of F() macros and really, less verbose screen prints since the same data is stored in file.

AVRs have no FPU. Floats work ssslllloooooowwwww of AVR-duinos. But that's about speed.

1 Like

Yes, I'm using an Uno, and I'm confused by why our school's tutorial "kit" code, with tons of sensors, takes 14% of the space on the Uno... but when I use sensors purchased from SparkFun and Adafruit, etc, that have all these libraries to contend with, suddenly I'm dealing with 60% space used up per sensor!

So, I'm not sure what I'm supposed to do since I have to use a bunch of sensors.

In the meantime, I enjoy that printCommaSpace function, and have incorporated it!

The only way I could get my code to work was to go away from character arrays and sprintf and to go back to basics, and just print each variable one by one.

I may try the array thing again later.

edit: ignore the following, I seem to have gotten it to work. I must've made an error. Thank you again!

I couldn't get the F() macro to work on a String, e.g.:

String fstr = (F("Hello"));
Serial.print(fstr);

I think that's what I did and it wasn't happyβ€”but maybe I should just avoid the string and use the F() instead, a la:

Serial.print(F("Hello"));

Thank you for your help!

So if we add the SD buffer with its 512 bytes - how much space you will have for your arrays? 100 bytes?
Certanly you have a memory problem.

You have to change the board to the one with more memory, i.e Arduino Mega or Arduino Nano Every

Not just about speed: it also means the AVR has to do all the FP calcs "manually" in code - so it does take up extra code space as well...

1 Like

Yep, that definitely sounds like you're running out of RAM.

Been there, done that.

sprintf is fat!

I don't use floats if possible. With unsigned long vars I have 9 digits of any unit I choose.
If I work in millimeters, I have meters to 3 places.
If I work in micrometers I can lose 3 places to lost remainders in division and STILL be exact to the millimeter.
If I need powers of 10, I'll use an int.
When I learned math, we had to write it down ... on clay tablets!

1 Like