Bare Bones Datalogging (UPDATE)

Hi,

So I couldn't get my head around SPI flash, so I give up and went back to the microsd card.

Implementing a few sleep techniques I managed to get it to run on the coin battery, but it only ran for half an hour. The coin battery had the capacity, but couldn't absorb the sudden writes to sd.

it sleeps at around 2ma and jumps to I think 15ma while writing to sd. I have a basic multimeter so I'm not sure if I'm catching the peak ma at write.

Anyhow I decided to add a small 3.3v regulator to the circuit and used a 100ma lipo battery, and hey presto, it worked. 24 hours of continuous accelerometer readings. Around 2 million data points. I'm over the moon with this because the lipo is only 20x20x3mm, which is about the same as the coin battery.

Now to see how much more I can get out of this. I have added my sketch below. Sorry its not commented, and I'm using two sleep libraries, so that needs tidied up a bit.

#include <LowPower.h>

#include <SD.h>
#include <Time.h>
#include <DSRTCLib.h>// RTC library
#include <Wire.h> // Must include Wire library for I2C
#include <SFE_MMA8452Q.h> // Includes the SFE_MMA8452Q library
#include <avr/power.h>
#include <avr/sleep.h>

int INT_PIN = 1; 
int int_number = 0; 
const int chipSelect = 10; 
MMA8452Q accel;

DS1339 RTC = DS1339(INT_PIN, int_number);

File file;

void setup()
{
  pinMode(INT_PIN, INPUT);
  digitalWrite(INT_PIN, HIGH);

  // enable deep sleeping
  set_sleep_mode(SLEEP_MODE_PWR_DOWN);
  sleep_enable();

  RTC.start();
  RTC.snooze(86400);
  
  Serial.begin(9600);
  pinMode(10, OUTPUT);
  accel.init();
  SD.begin(chipSelect);
  file = SD.open("datalog.txt", O_CREAT | O_WRITE);
}

void nap()
{
  // Dummy function. We don't actually want to do anything here, just use an interrupt to wake up.
  //RTC.clear_interrupt();
  // For some reason, sending commands to clear the interrupt on the RTC here does not work. Maybe Wire uses interrupts itself?
  Serial.print(".");
}

void loop()
{
  tmElements_t tm;

  if (file) 
  {
    for (uint8_t i = 0; i < 100; i++) {

      RTC.readTime();
      accel.read();

      print2digits(RTC.getHours());
      file.write(':');
      print2digits(RTC.getMinutes());
      file.write(':');
      print2digits(RTC.getSeconds());
      file.print('\t'); 
      printCalculatedAccels();
      printOrientation();
      file.println(); // Print new line every time.
      
      LowPower.powerDown(SLEEP_30MS, ADC_OFF, BOD_OFF); 
    }
    file.flush();
  }
}

void printCalculatedAccels()
{ 
  file.print(accel.cx, 3);
  file.print("\t");
  file.print(accel.cy, 3);
  file.print("\t");
  file.print(accel.cz, 3);
  file.print("\t");
}

void printOrientation()
{
  byte pl = accel.readPL();
  switch (pl)
  {
  case PORTRAIT_U:
    file.print("PU");
    break;
  case PORTRAIT_D:
    file.print("PD");
    break;
  case LANDSCAPE_R:
    file.print("LR");
    break;
  case LANDSCAPE_L:
    file.print("LL");
    break;
  case LOCKOUT:
    file.print("Fl");
    break;
  }
}

void print2digits(int number) {
  if (number >= 0 && number < 10) {
    file.write('0');
  }
  file.print(number);
}

It only writes to sd every 3 or 4 seconds and I believe it is this FOR loop that handles that

void loop()
{
  tmElements_t tm;

  if (file) 
  {
    for (uint8_t i = 0; i < 100; i++) {

      RTC.readTime();
      accel.read();

      print2digits(RTC.getHours());
      file.write(':');
      print2digits(RTC.getMinutes());
      file.write(':');
      print2digits(RTC.getSeconds());
      file.print('\t'); 
      printCalculatedAccels();
      printOrientation();
      file.println(); // Print new line every time.
      
      LowPower.powerDown(SLEEP_30MS, ADC_OFF, BOD_OFF); 
    }
    file.flush();
  }
}

Does that mean take 100 readings before flushing to sd? Whats the max readings it could take before flushing to sd? Where is the accelerometer data stored during this time considering the atmega is sleeping between reads?

I may in the future try and get interrupts from the accelerometer working.

For now, how far can I push that FOR loop? and how do I get the sd card to fully power down during sleep? At the min it uses 2ma with sd.h and sdfat.h.

Thanks for any advice you can give, and hope this helps someone else.

Cheers,

Damien

I have attached a picture of the schematic.

Do all my connections look ok?

Do I have all the proper capacitors and resistors? and do I need any more, or can I get rid of any?

Would appreciate any advice,

Cheers,

Damien.

I would suggest 0.1uf caps from ground to Avcc and Aref. Check the RTC and accel modules for pull up resistors. If the pull ups are present on one or both the module boards, you don't need the 2 10K external pull ups.

What model of battery are you using?

A write to the SD card occurs automatically whenever the 512 byte buffer fills. So assuming you are writing about 25 characters each pass through the loop, that would mean a block would be written to the card every 20 loops, or about once ever 0.6 seconds.

The flush call writes a partially filled buffer to the card and may update directory information as well. When the directory info is updated there are four block transfers (two reads and two writes). It's best not to flush too frequently. If you were to lose power or crash you would only lose data since the last flush.

My understanding is that SD cards go to sleep automatically when they are not being accessed and then draw a few hundred microamps. If you look at this thread you'll know as much about it as I do.

It is possible to disconnect the power to the SD card with a MOSFET. You can't use SD for that because it doesn't support reinitialization like SdFat does. There is also an energy hit that comes with initializing a card so it probably isn't worth it unless you are going to put it to sleep for a long time.

It's nice to see you making progress. I'm looking forward to a photo with the finished device on the back of a shrimp.

groundfungus:
I would suggest 0.1uf caps from ground to Avcc and Aref. Check the RTC and accel modules for pull up resistors. If the pull ups are present on one or both the module boards, you don't need the 2 10K external pull ups.

Hi,

Do you mean separate capacitors between vcc, Avcc and Aref. Or would one capacitor between power and ground cover them all?

The brake out board of the accelerometer has 4.7k pull-up resistors. I have its SCL/SDA lines paralleled with the DS1337's SCL/SDA lines, along with two 10k pull-up resistors for the DS1337. I am going to be making my own board, so just need 2 resistors to cover both devices. So 4.7k or 10k, but which one?

Cheers.

4.7K resistors are the usual pullups specified for I2C pullups, so I would go with those and just leave the 10K pullups off. A problem comes when there are too many pullups. If the parallel combination of all the pullup resistors on the I2C lines get to be to low, it becomes difficult to pull the lines low and communication fails. If the pullups are too high, noise can affect the lines and communication fails. Vcc, Avcc and Aref should each have a decoupling cap to bypass higher frequency noise to ground.

jboyton:
What model of battery are you using?

A write to the SD card occurs automatically whenever the 512 byte buffer fills. So assuming you are writing about 25 characters each pass through the loop, that would mean a block would be written to the card every 20 loops, or about once ever 0.6 seconds.

The flush call writes a partially filled buffer to the card and may update directory information as well. When the directory info is updated there are four block transfers (two reads and two writes). It's best not to flush too frequently. If you were to lose power or crash you would only lose data since the last flush.

My understanding is that SD cards go to sleep automatically when they are not being accessed and then draw a few hundred microamps. If you look at this thread you'll know as much about it as I do.

It is possible to disconnect the power to the SD card with a MOSFET. You can't use SD for that because it doesn't support reinitialization like SdFat does. There is also an energy hit that comes with initializing a card so it probably isn't worth it unless you are going to put it to sleep for a long time.

It's nice to see you making progress. I'm looking forward to a photo with the finished device on the back of a shrimp.

The batteries that I'm going to use in this first version are these https://www.hobbyking.com/hobbyking/store/uh_viewItem.asp?idProduct=37550. I can get smaller. but these ones were easily purchased.

It will be 10mm longer that I originally wanted by using these batteries, but I can hopefully make it thinner. as long as I can get everything crammed on the board. Hopefully this will not be an issue as I have been told that the "Nephrops norvegicus", the proper name for what I've been calling a shrimp, are actually larger than expected in Strangford Lough, which is where I'm from.

I've found the problem with the high sleep current of the card to be the card. Different cards seem to sleep a lot better than others.

I had worked that it would be a write to the sd card roughly every half second as well, but with that FOR loop at 100 it only seems to write every 4 seconds. as a test I increased the FOR loop to 200. It seemed to only write to the sd card every 8 seconds. I'm totally confused by this. I don't seem to be loosing any data. What do you recommend to be the best value for the FOR loop?

Cheers

groundfungus:
4.7K resistors are the usual pullups specified for I2C pullups, so I would go with those and just leave the 10K pullups off. A problem comes when there are too many pullups. If the parallel combination of all the pullup resistors on the I2C lines get to be to low, it becomes difficult to pull the lines low and communication fails. If the pullups are too high, noise can affect the lines and communication fails. Vcc, Avcc and Aref should each have a decoupling cap to bypass higher frequency noise to ground.

Cheers. So I'll use two 4.7k resistors to cover the I2C lines. I was kind of hoping you'd say one capacitor would cover all. All these capacitors and resistors take up an awful lot of space on a PCB :frowning:

Cheers.

The resistors on the modules should cover the required pullup resistance, so no need to add them to your board. It is much easier to design in the caps than to cobble them in later if the board acts funny.

groundfungus:
The resistors on the modules should cover the required pullup resistance, so no need to add them to your board. It is much easier to design in the caps than to cobble them in later if the board acts funny.

What do you mean the resistors on the modules? I'm not using a breakout board. I'll be designing my own from scratch.

Do you mean the accelerometer and DS1337 chips have internal resistors?

Cheers.

Seedler:
The batteries that I'm going to use in this first version are these https://www.hobbyking.com/hobbyking/store/uh_viewItem.asp?idProduct=37550. I can get smaller. but these ones were easily purchased.

That doesn't remotely match the description of the battery you gave in your first post: "a 100ma lipo battery... only 20x20x3mm, which is about the same as the coin battery."

Seedler:
...with that FOR loop at 100 it only seems to write every 4 seconds.

What do you mean "seems to write"? How are you sensing that the card is being written to?

jboyton:
That doesn't remotely match the description of the battery you gave in your first post: "a 100ma lipo battery... only 20x20x3mm, which is about the same as the coin battery."

What do you mean "seems to write"? How are you sensing that the card is being written to?

The 20x20x3mm batteries were from aliexpress. its an awful long delivery time, plus its really expensive to go for a quicker delivery. Apparently height is an issue too, so I thought two of these batteries could go either side of the board, which would have like a wrapping around saddle effect. Still trying to workout the best arrangement.

I have an led on the clock signal to the sd card. It does a double flash every 4 seconds, which corresponds to a jump in the ma reading from the multimeter, so I thought that was it writing to the sd card. Does that sound rite?

Cheers

Seedler:
I have an led on the clock signal to the sd card. It does a double flash every 4 seconds, which corresponds to a jump in the ma reading from the multimeter, so I thought that was it writing to the sd card. Does that sound rite?

It should write to the card every time the 512 byte buffer fills up. Where else are those data going to go?

You'll get a double flash on your led when you call flush(). The library has to update FAT to mark the blocks that were written to since the last flush. So it writes the current data block, fetches a block from FAT, modifies that and then writes it back, and then reads the current data block back into the buffer.

Maybe you just can't see the led flash when it's only writing a single block of data.

What do you mean the resistors on the modules? I'm not using a breakout board. I'll be designing my own from scratch.
Do you mean the accelerometer and DS1337 chips have internal resistors?

My bad. I made a misguided assumption. Even though your schematic shows the chips, not modules I got it wrong. I would go with the 4.7K resistors in place of the 10K resistors that you have. Sorry for my confusion.

jboyton:
It should write to the card every time the 512 byte buffer fills up. Where else are those data going to go?

You'll get a double flash on your led when you call flush(). The library has to update FAT to mark the blocks that were written to since the last flush. So it writes the current data block, fetches a block from FAT, modifies that and then writes it back, and then reads the current data block back into the buffer.

Maybe you just can't see the led flash when it's only writing a single block of data.

So it writes to the sd card when the buffer is full regardless of where in the loop it is?

Is there any need to use flush()? It's flush() that seems to cause the spike in the milliamps. Can I get rid of it?, or if not how often should I flush?

Is there any way to increase the 512 buffer? I see that the atmega328p has 2kb of ram, so does this mean the buffer could be three times larger than what it is now. Meaning less writes to SD, meaning longer logging time?

Cheers,

Damien

groundfungus:
My bad. I made a misguided assumption. Even though your schematic shows the chips, not modules I got it wrong. I would go with the 4.7K resistors in place of the 10K resistors that you have. Sorry for my confusion.

No prob, Thanks for your help.

Damien

Yes, it writes to the card when the buffer fills, at the point your code calls file.print().

There are advanced writing options supported by SdFat. You could research that. Another alternative would be to compress the information you are writing each time. You could easily cut it down by a factor of three although from previous discussions it seems you aren't interested in that. Post a question on the storage forum; hopefully you will attract the attention of user fat16lib who authored SdFat (and SD).

Unless you call flush (or close) your file will appear empty even though the data is on the card. So how many minutes of data loss would make you cry if your battery died unexpectedly? Pick some interval you'd be comfortable with and call flush that often. Maybe it's fifteen minutes, maybe it's five seconds. It's up to you.

jboyton:
Yes, it writes to the card when the buffer fills, at the point your code calls file.print().

There are advanced writing options supported by SdFat. You could research that. Another alternative would be to compress the information you are writing each time. You could easily cut it down by a factor of three although from previous discussions it seems you aren't interested in that. Post a question on the storage forum; hopefully you will attract the attention of user fat16lib who authored SdFat (and SD).

Unless you call flush (or close) your file will appear empty even though the data is on the card. So how many minutes of data loss would make you cry if your battery died unexpectedly? Pick some interval you'd be comfortable with and call flush that often. Maybe it's fifteen minutes, maybe it's five seconds. It's up to you.

Cool, thanks for that. I'll have a look at the storage forum and have a go at that.

I'll keep yous informed how I get on.

Cheers,

Damien

I did something similar a few years back, but I dropped the 328P to 8MHz so I could run everything at 3.3V (supplied by the regulator on the SD-Card breakout.)

Barebones 328 8Mhz ASCII datalogger

Ray

mrburnette:
I did something similar a few years back, but I dropped the 328P to 8MHz so I could run everything at 3.3V (supplied by the regulator on the SD-Card breakout.)

Barebones 328 8Mhz ASCII datalogger

Ray

Hi Ray,

Yeah I've got my 328p running at 8MHz too. It looks like you only had a couple of capacitors. Did everything work ok. My biggest problem is trying to get all the recommended resistors and capacitors ftted on the small board I'm designing.

Trying to route copper traces in EAGLE is a strange and frustrating puzzle :slight_smile:

Damien.