Go Down

Topic: Logging to SD file size limit? (Read 2 times) previous topic - next topic

D3C3PT1C0N

Nov 19, 2011, 07:18 pm Last Edit: Nov 19, 2011, 07:35 pm by D3C3PT1C0N Reason: 1
I have made a project that takes LDR / TEMPERATURE / DATE / TIME values and saves them to 2 Gb SD card.
All works well except the saving part which is glitchy -
logging stops when the file size on SD card is about 350 kb (or was it 350 b )

Has anyone had this kind of problem?

robtillaart

not encountered this, did some logging too for weeks continuously.
- what hardware are you using?
- please post the sketch - or a stripped version that still shows the bug.
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

D3C3PT1C0N

#2
Nov 19, 2011, 09:02 pm Last Edit: Nov 19, 2011, 09:40 pm by D3C3PT1C0N Reason: 1
First the hardware:

1) ATMEGA328P-PU microcontroller + all the stuff for breadboard arduino (capacitors, crystal)
2) SD Module (DFRobot.com)
3) RTC Module v1.1 (real time clock)
4) LM35 temp. sensor and LDR sensor

And my code:

Code: [Select]

#include <SD.h>
#include "Wire.h"
#define DS1307_I2C_ADDRESS 0x68
File file;
const int pin = 0;
int tempc = 0;
int ldr = 3;            
int ldr_value = 0;      
byte decToBcd(byte val)
{
 return ( (val/10*16) + (val%10) );
}
byte bcdToDec(byte val)
{
 return ( (val/16*10) + (val%16) );
}
void setDateDs1307(byte second,        // 0-59
                  byte minute,        // 0-59
                  byte hour,          // 1-23
                  byte dayOfWeek,     // 1-7
                  byte dayOfMonth,    // 1-28/29/30/31
                  byte month,         // 1-12
                  byte year)          // 0-99
{
  Wire.beginTransmission(DS1307_I2C_ADDRESS);
  Wire.send(0);
  Wire.send(decToBcd(second));  
  Wire.send(decToBcd(minute));
  Wire.send(decToBcd(hour));  
  Wire.send(decToBcd(dayOfWeek));
  Wire.send(decToBcd(dayOfMonth));
  Wire.send(decToBcd(month));
  Wire.send(decToBcd(year));
  Wire.endTransmission();
}
void getDateDs1307(byte *second,
         byte *minute,
         byte *hour,
         byte *dayOfWeek,
         byte *dayOfMonth,
         byte *month,
         byte *year)
{
 Wire.beginTransmission(DS1307_I2C_ADDRESS);
 Wire.send(0);
 Wire.endTransmission();

 Wire.requestFrom(DS1307_I2C_ADDRESS, 7);
 *second     = bcdToDec(Wire.receive() & 0x7f);
 *minute     = bcdToDec(Wire.receive());
 *hour       = bcdToDec(Wire.receive() & 0x3f);
 *dayOfWeek  = bcdToDec(Wire.receive());
 *dayOfMonth = bcdToDec(Wire.receive());
 *month      = bcdToDec(Wire.receive());
 *year       = bcdToDec(Wire.receive());
}

void setup() {
pinMode(13, OUTPUT);
byte second, minute, hour, dayOfWeek, dayOfMonth, month, year;
Wire.begin();
 Serial.begin(9600);

 second = 00;
 minute = 39;
 hour = 00;
 dayOfWeek = 6;
 dayOfMonth = 19;
 month = 11;
 year = 11;
//setDateDs1307(second, minute, hour, dayOfWeek, dayOfMonth, month, year);

if (!SD.begin()) {
  Serial.println("begin failed");
  return;
}}


void loop() {
byte second, minute, hour, dayOfWeek, dayOfMonth, month, year;

getDateDs1307(&second, &minute, &hour, &dayOfWeek, &dayOfMonth, &month, &year);

file = SD.open("TEST_SD.TXT", FILE_WRITE);

for(int i = 0; i < 8; i++) {tempc += analogRead(pin);}
tempc = (5.0 * tempc * 100.0) / (8.0 * 1024.0);

ldr_value = analogRead(ldr);

file.print(ldr_value);
file.print(" LDR ");
file.print(tempc,DEC);
file.print(" deg C ");
 file.print(hour, DEC);
 file.print(":");
 file.print(minute, DEC);
 file.print(":");
 file.print(second, DEC);
 file.print(" ");
 file.print(month, DEC);
 file.print("/");
 file.print(dayOfMonth, DEC);
 file.print("/");
 file.print(year, DEC);
 file.print(" Day_of_week:");
 file.println(dayOfWeek, DEC);

file.close();

analogWrite(9, 100);
 delay(1500);
 analogWrite(9, 0);

}


I have done a test where it logged temperature every 1 second - result: when file reached 301.8 Kb it stopped logging.
Also when i put that SD card to my laptop i didn't delete the log-file. When i plugged it in to Arduino, it continued to save in that file.

cyclegadget


  What is your power source? 

  Is this the LED to show a heart beat?
Code: [Select]
analogWrite(9, 100);
  delay(1500);
  analogWrite(9, 0);

D3C3PT1C0N

#4
Nov 21, 2011, 08:06 am Last Edit: Nov 21, 2011, 08:16 am by D3C3PT1C0N Reason: 1
Hello!
Power source is an old charger (maybe cellphone-s... ) which outputs 5.2V and 1A.
In the picture there are 7805 Voltage regulator + capacitor that are not actually in use... before i tried 9V battery but it got empty so i had to find another power source.

And it's not LED

Quote
analogWrite(9, 100);          // 9 - is ATMEGA328 pin where speaker is connected to..
 delay(1500);          // just how long the speaker makes sound
 analogWrite(9, 0);          // 0 - indicates silence or sound level


this makes a small SPEAKER make the heartbeat :)
(look at the picture...  there is no extra LED but only the speaker. Red led is from SD module that i cant control)

My script/code works quite well except that it stops recording when the log-file gets 300 kb or so...

D3C3PT1C0N

I still have the same problem - temperature logger stops saving to SD card after 200 - 300 kb. Arduino shows that it is working but nothing goes to SD card.

Maybe there is something with formatting the SD card... cant be any other limits

really really sad, that it doesnt work :(

robtillaart


Could not find the cause in your sketch, there might be a bug in the SD lib you use, in the append code as it must do an fseek to the end of the file. There might be an overflow in that as it seems to be quite reproducable. Can you post (zip attach) the SD lib you are using?

A workaround is to write one file per day or per month.

Can you write a small sketch that just dumps records as fast as possible to the SD card and see if it blocks also at ~300KB

If that blocks it would indicate that the SD is to blame.



Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

D3C3PT1C0N

Thank you for your ideas..
When i just make a LOOP for saving some text - it works well (DOES NOT stop after 300 kb).
I am using the default SD library that comes with Arduino IDE 022.

One solution could be to use another SD lib... although i dont think it is the problem, because TEXT saving worked..

Saving to separate file might be the best solution so far - for example after 300 kb new file is created...

hardcore

Try reformatting the card or using another one, before changing the library

D3C3PT1C0N

ok thanks! I will do it when i get back home.

fat16lib

I am amazed your setup works at all.  You need to clean up your wiring so the SD is close to the CPU and use very short wires for SPI.

The version of SD.h included with 0022 opens files with the O_SYNC flag so there is an incredible amount of I/O to the card. 

Also opening and closing the file for each data point causes more I/O as the file grows since open does a seek to the end-of-file.

I/O to the card is done in 512 byte blocks and a sync call requires at least 2048 byte of I/O.  I suspect you are doing over 50,000 bytes of I/O to the SD for each data point.

D3C3PT1C0N

#11
Feb 08, 2012, 07:15 pm Last Edit: Feb 08, 2012, 07:17 pm by D3C3PT1C0N Reason: 1
thank you for sharing your knowledge!
The main reason must be the open-close thing. Maybe making new file after 200 kb will solve the problem.
Or i can add a push-button that makes the file close.

ps. sorry for being so stupid.. but im playing with Arduino just for hobby

Go Up