Pages: [1]   Go Down
Author Topic: Logging to SD file size limit?  (Read 2132 times)
0 Members and 1 Guest are viewing this topic.
Europe, Estonia
Offline Offline
Full Member
***
Karma: 0
Posts: 208
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I have made a project that takes LDR / TEMPERATURE / DATE / TIME values and saves them to 2 Gb SD card.
All works well except the saving part which is glitchy -
logging stops when the file size on SD card is about 350 kb (or was it 350 b )

Has anyone had this kind of problem?
« Last Edit: November 19, 2011, 01:35:00 pm by D3C3PT1C0N » Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 216
Posts: 13673
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

not encountered this, did some logging too for weeks continuously.
- what hardware are you using?
- please post the sketch - or a stripped version that still shows the bug.
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Europe, Estonia
Offline Offline
Full Member
***
Karma: 0
Posts: 208
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

First the hardware:

1) ATMEGA328P-PU microcontroller + all the stuff for breadboard arduino (capacitors, crystal)
2) SD Module (DFRobot.com)
3) RTC Module v1.1 (real time clock)
4) LM35 temp. sensor and LDR sensor

And my code:

Code:
#include <SD.h>
#include "Wire.h"
#define DS1307_I2C_ADDRESS 0x68
File file;
const int pin = 0;
int tempc = 0;
int ldr = 3;            
int ldr_value = 0;      
byte decToBcd(byte val)
{
  return ( (val/10*16) + (val%10) );
}
byte bcdToDec(byte val)
{
  return ( (val/16*10) + (val%16) );
}
void setDateDs1307(byte second,        // 0-59
                   byte minute,        // 0-59
                   byte hour,          // 1-23
                   byte dayOfWeek,     // 1-7
                   byte dayOfMonth,    // 1-28/29/30/31
                   byte month,         // 1-12
                   byte year)          // 0-99
{
   Wire.beginTransmission(DS1307_I2C_ADDRESS);
   Wire.send(0);
   Wire.send(decToBcd(second));  
   Wire.send(decToBcd(minute));
   Wire.send(decToBcd(hour));  
   Wire.send(decToBcd(dayOfWeek));
   Wire.send(decToBcd(dayOfMonth));
   Wire.send(decToBcd(month));
   Wire.send(decToBcd(year));
   Wire.endTransmission();
}
void getDateDs1307(byte *second,
          byte *minute,
          byte *hour,
          byte *dayOfWeek,
          byte *dayOfMonth,
          byte *month,
          byte *year)
{
  Wire.beginTransmission(DS1307_I2C_ADDRESS);
  Wire.send(0);
  Wire.endTransmission();

  Wire.requestFrom(DS1307_I2C_ADDRESS, 7);
  *second     = bcdToDec(Wire.receive() & 0x7f);
  *minute     = bcdToDec(Wire.receive());
  *hour       = bcdToDec(Wire.receive() & 0x3f);
  *dayOfWeek  = bcdToDec(Wire.receive());
  *dayOfMonth = bcdToDec(Wire.receive());
  *month      = bcdToDec(Wire.receive());
  *year       = bcdToDec(Wire.receive());
}

void setup() {
 pinMode(13, OUTPUT);
byte second, minute, hour, dayOfWeek, dayOfMonth, month, year;
Wire.begin();
  Serial.begin(9600);

  second = 00;
  minute = 39;
  hour = 00;
  dayOfWeek = 6;
  dayOfMonth = 19;
  month = 11;
  year = 11;
//setDateDs1307(second, minute, hour, dayOfWeek, dayOfMonth, month, year);
 
 if (!SD.begin()) {
   Serial.println("begin failed");
   return;
 }}


void loop() {
byte second, minute, hour, dayOfWeek, dayOfMonth, month, year;

getDateDs1307(&second, &minute, &hour, &dayOfWeek, &dayOfMonth, &month, &year);

file = SD.open("TEST_SD.TXT", FILE_WRITE);

for(int i = 0; i < 8; i++) {tempc += analogRead(pin);}
tempc = (5.0 * tempc * 100.0) / (8.0 * 1024.0);

ldr_value = analogRead(ldr);

file.print(ldr_value);
file.print(" LDR ");
file.print(tempc,DEC);
file.print(" deg C ");
  file.print(hour, DEC);
  file.print(":");
  file.print(minute, DEC);
  file.print(":");
  file.print(second, DEC);
  file.print(" ");
  file.print(month, DEC);
  file.print("/");
  file.print(dayOfMonth, DEC);
  file.print("/");
  file.print(year, DEC);
  file.print(" Day_of_week:");
  file.println(dayOfWeek, DEC);

file.close();

analogWrite(9, 100);
  delay(1500);
  analogWrite(9, 0);

}

I have done a test where it logged temperature every 1 second - result: when file reached 301.8 Kb it stopped logging.
Also when i put that SD card to my laptop i didn't delete the log-file. When i plugged it in to Arduino, it continued to save in that file.
« Last Edit: November 19, 2011, 03:40:39 pm by D3C3PT1C0N » Logged

Greenville, IL
Offline Offline
Edison Member
*
Karma: 15
Posts: 1330
Warning Novice on board! 0 to 1 chance of errors!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


  What is your power source? 

  Is this the LED to show a heart beat?
 
Code:
analogWrite(9, 100);
  delay(1500);
  analogWrite(9, 0);
Logged


Europe, Estonia
Offline Offline
Full Member
***
Karma: 0
Posts: 208
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hello!
Power source is an old charger (maybe cellphone-s... ) which outputs 5.2V and 1A.
In the picture there are 7805 Voltage regulator + capacitor that are not actually in use... before i tried 9V battery but it got empty so i had to find another power source.

And it's not LED

Quote
analogWrite(9, 100);          // 9 - is ATMEGA328 pin where speaker is connected to..
  delay(1500);          // just how long the speaker makes sound
  analogWrite(9, 0);          // 0 - indicates silence or sound level

 this makes a small SPEAKER make the heartbeat smiley
(look at the picture...  there is no extra LED but only the speaker. Red led is from SD module that i cant control)

My script/code works quite well except that it stops recording when the log-file gets 300 kb or so...
« Last Edit: November 21, 2011, 02:16:51 am by D3C3PT1C0N » Logged

Europe, Estonia
Offline Offline
Full Member
***
Karma: 0
Posts: 208
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I still have the same problem - temperature logger stops saving to SD card after 200 - 300 kb. Arduino shows that it is working but nothing goes to SD card.

Maybe there is something with formatting the SD card... cant be any other limits

really really sad, that it doesnt work smiley-sad
Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 216
Posts: 13673
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


Could not find the cause in your sketch, there might be a bug in the SD lib you use, in the append code as it must do an fseek to the end of the file. There might be an overflow in that as it seems to be quite reproducable. Can you post (zip attach) the SD lib you are using?

A workaround is to write one file per day or per month.

Can you write a small sketch that just dumps records as fast as possible to the SD card and see if it blocks also at ~300KB

If that blocks it would indicate that the SD is to blame.



 
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Europe, Estonia
Offline Offline
Full Member
***
Karma: 0
Posts: 208
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thank you for your ideas..
When i just make a LOOP for saving some text - it works well (DOES NOT stop after 300 kb).
I am using the default SD library that comes with Arduino IDE 022.

One solution could be to use another SD lib... although i dont think it is the problem, because TEXT saving worked..

Saving to separate file might be the best solution so far - for example after 300 kb new file is created...
Logged

Offline Offline
Jr. Member
**
Karma: 0
Posts: 98
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Try reformatting the card or using another one, before changing the library
Logged

Europe, Estonia
Offline Offline
Full Member
***
Karma: 0
Posts: 208
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

ok thanks! I will do it when i get back home.
Logged

0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1631
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I am amazed your setup works at all.  You need to clean up your wiring so the SD is close to the CPU and use very short wires for SPI.

The version of SD.h included with 0022 opens files with the O_SYNC flag so there is an incredible amount of I/O to the card. 

Also opening and closing the file for each data point causes more I/O as the file grows since open does a seek to the end-of-file.

I/O to the card is done in 512 byte blocks and a sync call requires at least 2048 byte of I/O.  I suspect you are doing over 50,000 bytes of I/O to the SD for each data point.
Logged

Europe, Estonia
Offline Offline
Full Member
***
Karma: 0
Posts: 208
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

thank you for sharing your knowledge!
The main reason must be the open-close thing. Maybe making new file after 200 kb will solve the problem.
Or i can add a push-button that makes the file close.

ps. sorry for being so stupid.. but im playing with Arduino just for hobby
« Last Edit: February 08, 2012, 01:17:39 pm by D3C3PT1C0N » Logged

Pages: [1]   Go Up
Jump to: