Increasing SD library buffer size

Hi,

Is it possible to increase the SD library buffer size from 512 bytes to 1024 bytes?

The 512 buffer fills with accelerometer data and writes to micro sd every 0.6 seconds. I then call flush() roughly every 15 mins.

With an 80mah single cell lipo I get roughly 50 hours of logging. I was thinking if I could increase the buffer and half the amount of times I wrote to the sd card, I could roughly double battery life.

Does anyone know if this is possible?

Here is a copy of my code:

#include <LowPower.h>

#include <SD.h>
#include <Time.h>
#include <DSRTCLib.h>// RTC library
#include <Wire.h> // Must include Wire library for I2C
#include <SFE_MMA8452Q.h> // Includes the SFE_MMA8452Q library
#include <avr/power.h>
#include <avr/sleep.h>

int INT_PIN = 1; 
int int_number = 0; 
const int chipSelect = 10; 
MMA8452Q accel;

DS1339 RTC = DS1339(INT_PIN, int_number);

File file;

void setup()
{
  pinMode(INT_PIN, INPUT);
  digitalWrite(INT_PIN, HIGH);

  // enable deep sleeping
  set_sleep_mode(SLEEP_MODE_PWR_DOWN);
  sleep_enable();

  RTC.start();
  RTC.snooze(86400);
  
  Serial.begin(9600);
  pinMode(10, OUTPUT);
  accel.init(SCALE_2G, ODR_50);
  SD.begin(chipSelect);
  file = SD.open("datalog.txt", O_CREAT | O_WRITE);
}

void nap()
{
  // Dummy function. We don't actually want to do anything here, just use an interrupt to wake up.
  //RTC.clear_interrupt();
  // For some reason, sending commands to clear the interrupt on the RTC here does not work. Maybe Wire uses interrupts itself?
  Serial.print(".");
}

void loop()
{
  tmElements_t tm;

  if (file) 
  {
    for (uint8_t i = 0; i < 20000; i++) {

      RTC.readTime();
      accel.read();

      print2digits(RTC.getHours());
      file.write(':');
      print2digits(RTC.getMinutes());
      file.write(':');
      print2digits(RTC.getSeconds());
      file.print('\t'); 
      printCalculatedAccels();
      printOrientation();
      file.println(); // Print new line every time.
      
      LowPower.powerDown(SLEEP_30MS, ADC_OFF, BOD_OFF); 
    }
    file.flush();
  }
}

void printCalculatedAccels()
{ 
  file.print(accel.cx, 3);
  file.print("\t");
  file.print(accel.cy, 3);
  file.print("\t");
  file.print(accel.cz, 3);
  file.print("\t");
}

void printOrientation()
{
  byte pl = accel.readPL();
  switch (pl)
  {
  case PORTRAIT_U:
    file.print("PU");
    break;
  case PORTRAIT_D:
    file.print("PD");
    break;
  case LANDSCAPE_R:
    file.print("LR");
    break;
  case LANDSCAPE_L:
    file.print("LL");
    break;
  case LOCKOUT:
    file.print("Fl");
    break;
  }
}

void print2digits(int number) {
  if (number >= 0 && number < 10) {
    file.write('0');
  }
  file.print(number);
}

Cheers,

Damien

Be aware that a bug in SD.h prevents SD cards from sleeping at low current. SD.h is a wrapper for a very old version of SdFat. The problem was fixed in SdFat about four years ago.

SD cards in SPI mode will not sleep unless you send a byte on the SPI bus after SD chip select goes high. Here is the current code in SdFat.

  digitalWrite(m_chipSelectPin, HIGH);
  // insure MISO goes high impedance
  spiSend(0XFF);

The 512 byte buffer is a block cache for directory entries, FAT entries, and data for all open files. It is not associated with a file so you can't change the size.

SD cards take high current to program flash and the amount of power used depends on how much data you write. SD cards draw little current when idle so increasing the buffer won't help much.

Typical currents are 80 ma write, 60 ma read, 0.2 ma standby, 0.1 ma sleep for an industrial SD like this.

Hi, Thanks for the reply.

I was getting pretty good sleep currents using the SD library. Overall sleep was 0.17ma. I tried implementing the sdFat library and got the exact same results. I have found that the SD card makes a huge difference, with some upto 10ma while sleeping.

Thanks for letting me know about the buffer.

I was wondering if you could help me with this bit of code:

void loop()
{
  tmElements_t tm;

  if (file) 
  {
    for (uint8_t i = 0; i < 200; i++) {

      RTC.readTime();
      accel.read();

      print2digits(RTC.getHours());
      file.write(':');
      print2digits(RTC.getMinutes());
      file.write(':');
      print2digits(RTC.getSeconds());
      file.print('\t'); 
      printCalculatedAccels();
      printOrientation();
      file.println(); // Print new line every time.
      
      LowPower.powerDown(SLEEP_30MS, ADC_OFF, BOD_ON); 
    }
    file.flush();
  }
}

It writes to SD every 0.6 seconds and then calls flush() every 200 loops, which works out to be about every 7 seconds. When flush() is called you can see a definite jump in ma on the multimeter.

I was hoping to extend battery life by calling flush every 15 mins, so I set the loop to 20,000. Nothing saved. I have found that anything over 200 loops wont save anything to the sd card.

So my question is. Would calling flush() less often save on battery life, and if so how to I call it at a longer interval? And if the while loop is the best option, what is the best number for the loop?

Cheers,

Damien.

I have found that the SD card makes a huge difference, with some upto 10ma while sleeping.

The big variation is likely due to the MISO problem. Some cards go high Z and sleep without the byte of output.

SD.h is just an old SdFat without the byte of output.

Would calling flush() less often save on battery life,

Not much but you only need to call flush to insure the directory entry is updated.

Data is written to the SD when necessary but the directory entry is not updated until flush or close is called. If the file is not closed, you will only see data up to the last flush call.

You could save power by writing a binary file. It would require many fewer byte, just four bytes for date/time in time_t format. You also save CPU cycles formatting strings and slow character I/O which requires power.

So it would still be best to use sdFat.h?

I just added this to my sketch and removed SD.h:

#include <SdFat.h>
SdFat SD;

Is that all that’s needed?

I cant get my head around saving in binary, but I’ll have a look into it.

So if the output file is in binary, how do I converted it back into usable data?

Cheers,

Damien

So it would still be best to use sdFat.h?

It probably won't matter. If you need to log data at 30 Hz, you need a whole new structure for a data logger.

Your time jitter between points is probably greater than the 35 ms you sleep. Write latency to a file system on an SD can vary greatly. Worst case can be over 200 ms when searching for free clusters or when the SD moves data for wear-leveling.

You can look at any number of website for the theory of measuring time series. Think of a graph, the x and t values need similar accuracy or it is not worth logging at a given rate.

If you measure data values to 1%, time jitter must be 1%. Under 0.3 ms in your case.

fat16lib:
It probably won't matter. If you need to log data at 30 Hz, you need a whole new structure for a data logger.

Your time jitter between points is probably greater than the 35 ms you sleep. Write latency to a file system on an SD can vary greatly. Worst case can be over 200 ms when searching for free clusters or when the SD moves data for wear-leveling.

You can look at any number of website for the theory of measuring time series. Think of a graph, the x and t values need similar accuracy or it is not worth logging at a given rate.

If you measure data values to 1%, time jitter must be 1%. Under 0.3 ms in your case.

I'm looking to log around 20Hz. I choose the 30ms sleep because it was one of the sleep options closest to what I wanted. It gives me around 25 readings per second, which is more than enough. I haven't checked every second recorded for exact datapoints, but It looks to be at least over 20 per second. I'm happy with the speed that I logs. I'm just trying to squeeze a bit more life out of the batteries now.

Cheers,

Damien.

OK but why do you want to waste time recording worthless data? You have no idea about the time distribution of your 25 points.

Why are you recording the data?

I think you are using an ADXL362. This chip can record data at 25 Hz to a 512 sample FIFO. The FIFO can store about 170 data points, each with three axis data. You could sleep for several seconds then read the FIFO. This will minimize jitter in sample interval and save power.

You should create a large contiguous file and write binary data in raw mode. See the SdFat LowLatencyLogger example.

Do you mean recording the same time 25 times? I'm still green at programming. I just fumbling my way through snippets of code to try and get it to do what I want.

I would be saving time only once per 25 accelerometer readings, but haven't worked that out yet.

What to you mean about time distribution? Will the RTC not tell me when each data point was logged?

This is for a friend who wants to record accelerometer movement of a small lobster like creature. she wants to see if she can see different behaviours at different times, which is why she wants RTC data as well.

Cheers,

Damien.

You don't need the RTC for every point since there are about 25 points per second.

I would wake the CPU with the RTC SQW/INT signal once a second. I would record the time and the 25 points from the accelerometer FIFO. The interval between points will be fairly equal since the accelerometer clock won't drift too much.

You would only generate a little over 150 bytes per second. Three bytes of RTC data plus 25 accelerometer samples of six bytes each. If 8-bit acceleration was enough, only about 80 bytes per second.

This is probably a bit too much if you are new to this type programming. This would allow saving power.

I won't go into buffering data and avoiding busy waits on the SD. The LowLatencyLogger example demonstrates this.

Another note. You probably see low sleep current for SD cards with SD.h since you access the SPI bus for the accelerometer. This access frees the SD to go into sleep mode.

Thanks for clarifying that. I think binary would be my best way to go if I can work it out.

I already use INTA of RTC to wake the CPU. I initially let it sleep for 24 hours before recording. Is that where I would see the benefit of sdFat over SD?

I think it takes about 1/4 of a second to take the 25 readings. That means a good portion of the second would not be recorded. I would need to have the 25 readings spaced evenly over the second so as not to miss any subtle movements. that’s why I thought a small sleep between recordings would space it out better plus save a little power.

The accelerometer is 12 bit, and both it and the RTC are I2C devices.

Thanks for your help. I let you know how I get on,

cheers,

Damien.

I don't seem to explain this very well.

You don't want to wake the CPU 25 times a second. Most modern accelerometers are programmable for rate. You can setup for 25 samples per second to be stored in a FIFO. With a typical FIFO you can store six or seven seconds of data.

Typical accelerometers have a 512 deep FIFO with each entry storing one 12 bit value. This would be 170 sample of three axis in your case or more than six seconds.

You wake the CPU with the RTC and read the FIFO. You write a large block of binary data. You use a large pre-allocated file.

This will save a huge amount of power.

I have worked with users who have logged data for nearly a year using three AA batteries, about 2000mah. Their rates were much lower but you should be able to extend your time a great deal with 80 mah at your rates.

You don't want to spin in a busy loop while the accelerometers digitizes data you are wasting a huge amount of CPU time when you could be sleeping.

I didn't know you could do that. Sadly my MMA8452Q doesn't have FIFO. Might look into one that does.

Cheers,

Damien