Need help understanding Buffer methods to limit SD card writes

Hi,

I'm fairly new to C and am trying to make a long-life gps logger from a tiny-duino. I am having some issues optimising some example code given out by Make Magazine : GPS Cat Tracker - Make:

This code appears to be writing each character to the SD card one at a time, resulting in many writes each second. I will be taking this GPS Logger overseas with me as I have done in the past with commercial devices and need to optimise it for fpower-consumption so I am not lugging a massive battery around that goes flat in 50 minutes from writing to an SD card thousands of times.

If possible, I want to store this data temporarily and then write in bursts to keep the writes to a minimum. Many SD Data logger projects on this site focus on speed, and trying to get as many readings to the card in the shortest time possible so this project changes things a little, with a data point only needed once a second. I will most likely change this to once every couple of seconds to conserve power even further.

Has anyone done this before with the goal being to limit power consumption on SD writes? I am unsure on how to store this data on the 328P temporarily and then call it to write. Write speed isn't an issue with this project as the next reading won't need to go for 1 or more seconds.

At the moment utilising this code and a 2200mAh 18650 Li-Ion Battery, I am getting approximately 43 hours worth of logged data points. The commercial version I have is the Qstarz bt-q1000xt and was fine for my previous holidays, getting 40hours and extremely fast gps locks, even indoors, from a 1000mAh battery, but requires me to use a computer to retrieve the data, which fills in about 80hours worth of logging. This holiday to Europe in a few months will be on a tour and taking a laptop to retrieve data will suck bigtime. Being able to simply write the data to a micro SD allows for a metric shite-load of data to be stored, as well as the potential to simply switch it out (after ~3 months of data).

Some direction from more experienced programmers would be appreciated.

pickle_64:
I am unsure on how to store this data on the 328P temporarily

The data can be stored in variables in the Arduino SRAM for as long as you want with the risk of data loss if the power fails.

Writing to the Arduino EEPROM is probably less attractive than to the SD Card because of the limited total number of writes to the EEPROM.

...R

Would i store byte by byte into lots of different variables, or can i store multiple bytes as 1 object?

e.g. an NMEA output from the GPS Module:

$GPRMC,092750.000,A,5321.6802,N,00630.3372,W,0.02,31.66,280511,,,A43
$GPGGA,092751.000,5321.6802,N,00630.3371,W,1,8,1.03,61.7,M,55.3,M,,75
$GPGSA,A,3,10,07,05,02,29,04,08,13,,,,,1.72,1.03,1.38
0A
$GPGSV,3,1,11,10,63,137,17,07,61,098,15,05,59,290,20,08,54,157,30
70
$GPGSV,3,2,11,02,39,223,16,13,28,070,17,26,23,252,,04,14,186,15*77
$GPGSV,3,3,11,29,09,301,24,16,09,020,,36,,,*76

Would I save '$', 'G', 'P', 'R', 'M', 'C', etc... or can I take in the bytes of the NMEA output as a massive string?

With the tinyduino using a 328P, is 2kB RAM really enough to implement this idea? It also has 32kB of flash memory but I don't know how to utilise this memory instead of generic RAM. Sorry, but lots of questions at the moment.

The flash memory is not available at run time so you are stuck with the 2k of SRAM. How much can be used to store GPS data depends on other features of your program.

You can store the data from the GPS as a string that is a char array with 0 as the value of the character after the end of the data. Do NOT try to use Strings (with a capital S) - they don't work well with the limited SRAM in an Arduino - especially if you need to use nearly all of the SRAM.

The Arduino code in this demo shows how the incoming data is saved in a char array as it arrives.

If you have enough memory you could organize a buffer as an array of char arrays with, perhaps, 3 or 5, char arrays. Then as data is received it can be save it in the next array. When all the arrays have data you could write them to the SD card and then start over.

I don't know whether you need to interpret the GPS data on the Arduino. Obviously that would involve a certain amount of memory "duplication". For example you would have the lattitude within the string received and in a variable. This will have an impact on how much memory is available to store raw GPS data.

...R

Hi again,

An update of where the project is up to at the moment:

/*
   This Arduino sketch will log GPS NMEA data to a SD card every second
*/

#include <SoftwareSerial.h>
#include <SD.h>
#include <avr/sleep.h>
#include <avr/power.h>


// The Arduino pins used by the GPS module
static const int GPS_ONOFFPin = A3;
static const int GPS_SYSONPin = A2;
static const int GPS_RXPin = A1;
static const int GPS_TXPin = A0;
static const int GPSBaud = 9600;
static const int chipSelect = 10;
static const int buffsize = 200;
volatile int f_timer=0;


// The GPS connection is attached with a software serial port
SoftwareSerial Gps_serial(GPS_RXPin, GPS_TXPin);





////////////////////////////////

void enterSleep(void)
{
  set_sleep_mode(SLEEP_MODE_IDLE);
  
  sleep_enable();


  /* Disable all of the unused peripherals. This will reduce power
   * consumption further and, more importantly, some of these
   * peripherals may generate interrupts that will wake our Arduino from
   * sleep!
   */
  //power_adc_disable();
  //power_spi_disable();
  power_timer0_disable();
  power_timer2_disable();
  power_twi_disable();  

  /* Now enter sleep mode. */
  sleep_mode();
  
  /* The program will continue from here after the timer timeout*/
  sleep_disable(); /* First thing to do is disable sleep. */
  
  /* Re-enable the peripherals. */
  power_all_enable();
}

///////////////////////////////
ISR(TIMER1_OVF_vect)
{
  /* set the flag. */
   if(f_timer == 0)
   {
     f_timer = 1;
   }
}
/////////////////////////////////



void setup()
{    
  // Init the GPS Module to wake mode
  pinMode(GPS_SYSONPin, INPUT);
  pinMode(GPS_ONOFFPin, OUTPUT);
  digitalWrite( GPS_ONOFFPin, LOW );   
  delay(5); 
  if( digitalRead( GPS_SYSONPin ) == LOW )
  {
     // Need to wake the module
    digitalWrite( GPS_ONOFFPin, HIGH ); 
    delay(5); 
    digitalWrite( GPS_ONOFFPin, LOW );      
  } 
     
  ////*** Configure the timer.***////
  
  /* Normal timer operation.*/
  TCCR1A = 0x00; 
  
  /* Clear the timer counter register.
   * You can pre-load this register with a value in order to 
   * reduce the timeout period, say if you wanted to wake up
   * ever 4.0 seconds exactly.
   */
  TCNT1=0x0000; 
  
  /* Configure the prescaler for 1:1024, giving us a 
   * timeout of 4.09 seconds.
   */
  TCCR1B = 0x04;
  
  /* Enable the timer overlow interrupt. */
  TIMSK1=0x01;
  
  /////          /////
  
  // Open the debug serial port at 9600
  Serial.begin(9600);
 
  // Open the GPS serial port  
  Gps_serial.begin(GPSBaud);  
   
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");   
}


int inByte = 0;         // incoming serial byte
byte pbyGpsBuffer[buffsize];
int byBufferIndex = 0;


//////////////////////////
void loop()
{
  if(f_timer==1)
  {
    f_timer = 0;
    Serial.print("Timer1 overflow");
    SDCard();
    
    /* Re-enter sleep mode. */
    enterSleep();
  }
}
///////////////////////

void SDCard()
{
  byte byDataByte = 0;
  
  //A delay here is how long the gap 
  //between writing each char to the 
  //serial port is. e.g. Delay(1000) 
  //would give 1 char per second
  
  if (Gps_serial.available())
  
  {
     byDataByte = Gps_serial.read();
    
     //Serial.write(byDataByte);
     pbyGpsBuffer[ byBufferIndex++ ] = byDataByte;
     
     if( byBufferIndex >= buffsize )
     {
       byBufferIndex = 0;       
       File dataFile = SD.open("gps.txt", FILE_WRITE);
    
       // if the file is available, write to it:
       if (dataFile) {
        dataFile.write(pbyGpsBuffer, buffsize);
        dataFile.close();
        Serial.write(pbyGpsBuffer, buffsize);
      }  
      // if the file isn't open, pop up an error:
      else {
        Serial.println("error opening gps.txt");
      }        
     }      
  }
}

The program has sleep integrated into it with a wakeup set for timer 1. Ideally this will change to be an external interrupt pin wake-up by connecting a resistor between the pins used by the gps module to communicate with the board and the external interrupt pin. I'm going to have troubl working that into a board so small though. This will need to be implemented as the code currently relies on it waking up at the same moment the gps becomes available in order to track a point, which means that about 1 in 20 points gets recorded. At the moment, the breakthrough is getting sleep implemented and functional.

I also don't know at which levels of sleep the gps will continue to function. I still have yet to find out how to change the module to a lesser power state. Apparently it has high power mode at thirty something mA and an idle mode at ~10mA which gives only one point per second, which is more than enough.

I had a look at the demo you sent which interprets the incoming chars and builds its own array and think this is what will need to be implemented into the code:

//=============

void getDataFromPC() {

    // receive data from PC and save it into inputBuffer
    
  if(Serial.available() > 0) {

    char x = Serial.read();

      // the order of these IF clauses is significant
      
    if (x == endMarker) {
      readInProgress = false;
      newDataFromPC = true;
      inputBuffer[bytesRecvd] = 0;
      parseData();
    }
    
    if(readInProgress) {
      inputBuffer[bytesRecvd] = x;
      bytesRecvd ++;
      if (bytesRecvd == buffSize) {
        bytesRecvd = buffSize - 1;
      }
    }

    if (x == startMarker) { 
      bytesRecvd = 0; 
      readInProgress = true;
    }
  }
}

//=============
 
void parseData() {

    // split the data into its parts
    
  char * strtokIndx; // this is used by strtok() as an index
  
  strtokIndx = strtok(inputBuffer,",");      // get the first part - the string
  strcpy(messageFromPC, strtokIndx); // copy it to messageFromPC
  
  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  newFlashInterval = atoi(strtokIndx);     // convert this part to an integer
  
  strtokIndx = strtok(NULL, ","); 
  servoFraction = atof(strtokIndx);     // convert this part to a float

}

//=============

This will need to be implemented as when the gps is available and wakes up the board to take a reading and write, it keeps going until it is no longer available, which isn't neccessarily at the end of a point. So when writing to the serial port to identify the char arrays that got sent over, it bursts maybe 1.5 data points. By interpretting each byte in the code above, it could be modified to understand that every 6th "\n" is the end of a reading and to send that before attempting another read, or just going straight back to sleep.

When I tried this initially, i received an error regarding an illegal conversion from a char to something called a const uint8_*t(?).

Any ideas why trying to write the interpretted array to the SD card via

 dataFile.write(pbyGpsBuffer, buffsize);

would result in this error? Sorry but I don't have the adapted code handy. I lent it to a mate that dabbles in this area to have a look over.

I think it's just a case of trying to save char values where the code expects to see byte values (uint8). Either use a cast (byte) or change the data type in my code.

...R

OK!

I think I have it sorted now (compiles, but can't test as I still have to short 2 pins together):

/*
   This Arduino sketch will log GPS NMEA data to a SD card every second
*/

#include <SoftwareSerial.h>
#include <SD.h>
#include <avr/sleep.h>
#include <avr/power.h>


// The Arduino pins used by the GPS module
static const int GPS_ONOFFPin = A3;
static const int GPS_SYSONPin = A2;
static const int GPS_RXPin = A1;
static const int GPS_TXPin = A0;
static const int GPSBaud = 9600;
static const int chipSelect = 10;
const byte buffSize = 200;
volatile int f_timer=0;
const char startMarker = '

The line where I was getting the illegal conversion was

dataFile.write(inputBuffer, sendLength);

and when I couldn't get around this, I changed it to

dataFile.write(inputBuffer);

Which I hope will work.

I haven't tried it yet as i am still looking for a way to short the pins so that as soon as the GPS module starts giving out data, the interrupt off pin 2 is activated and wakes the system up.

It has the beginning marker as the dollar sign that each NMEA line begins with, and the ending marker as \n. This means it will only be taking one line per write at the moment which is not what I want, but is a good step forward in regulating the data to be written. I know that the NMEA data has 6 lines, so could easily solve this by changing the main loop to :

void loop()
{
    getDataFromGPS();
    SDCard();   

    getDataFromGPS();
    SDCard();   

    getDataFromGPS();
    SDCard();   

    getDataFromGPS();
    SDCard();   

    getDataFromGPS();
    SDCard();   

    getDataFromGPS();
    SDCard();   
    
    /* Re-enter sleep mode. */
    enterSleep();
}

but 6 writes is very inefficient. I'll sort this out after testing this current method works after shorting these pins. Any advice for shorting pins on micro boards?;
const char endMarker = '\n';
boolean readInProgress = false;
boolean newDataFromGPS = false;
byte bytesRecvd = 0;
char inputBuffer[buffSize];
char messageFromGPS[buffSize] = {0};
int pin2 = 2;
byte sendLength;

// The GPS connection is attached with a software serial port
SoftwareSerial Gps_serial(GPS_RXPin, GPS_TXPin);

////////////////////////////////

void enterSleep(void)
{
  set_sleep_mode(SLEEP_MODE_IDLE);
 
  sleep_enable();

/* Setup pin2 as an interrupt and attach handler. */
  attachInterrupt(0, pin2Interrupt, LOW);
  delay(100);
 
  set_sleep_mode(SLEEP_MODE_IDLE);
 
 
  power_timer0_disable();
  power_timer2_disable();
  power_twi_disable();

/* Now enter sleep mode. /
  sleep_mode();
 
  /
The program will continue from here after the timer timeout*/
  sleep_disable(); /* First thing to do is disable sleep. /
 
  /
Re-enable the peripherals. */
  power_all_enable();
}

////////////////////////////////

void pin2Interrupt(void)
{
  /* This will bring us back from sleep. /
 
  /
We detach the interrupt to stop it from
  * continuously firing while the interrupt pin
  * is low.
  */
  detachInterrupt(0);
}

////////////////////////////////

void setup()
{   
  // Init the GPS Module to wake mode
  pinMode(GPS_SYSONPin, INPUT);
  pinMode(GPS_ONOFFPin, OUTPUT);
  digitalWrite( GPS_ONOFFPin, LOW ); 
  delay(5);
  if( digitalRead( GPS_SYSONPin ) == LOW )
  {
    // Need to wake the module
    digitalWrite( GPS_ONOFFPin, HIGH );
    delay(5);
    digitalWrite( GPS_ONOFFPin, LOW );     
  }
   
  ///////////////* Setup the pin direction. */////////////////////
  pinMode(pin2, INPUT);
 
  ////////////////////////////////////////////////////////////////
 
  // Open the debug serial port at 9600
  Serial.begin(9600);

// Open the GPS serial port 
  Gps_serial.begin(GPSBaud); 
 
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
 
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized."); 
}

int inByte = 0;        // incoming serial byte
byte pbyGpsBuffer[buffSize];
int byBufferIndex = 0;

//////////////////////////

void loop()
{
    getDataFromGPS();
    SDCard(); 
   
    /* Re-enter sleep mode. */
    enterSleep();
}

///////////////////////

void SDCard()
{
    File dataFile = SD.open("gps.txt", FILE_WRITE);
   
    // if the file is available, write to it:
    if (dataFile)
    {
      //sendLength = (char) bytesRecvd;
      dataFile.write(inputBuffer);
      dataFile.close();
      Serial.write(inputBuffer);
    } 
      // if the file isn't open, pop up an error:
    else
    {
      Serial.println("error opening gps.txt");
    }     
}

//=============

void getDataFromGPS() {

// receive data from PC and save it into inputBuffer
   
  if(Serial.available() > 0) {

char x = Serial.read();

// the order of these IF clauses is significant
     
    if (x == endMarker) {
      inputBuffer[bytesRecvd] = x;
      bytesRecvd ++;
      readInProgress = false;
      newDataFromGPS = true;
      inputBuffer[bytesRecvd] = 0;
      return;
    }
   
    if(readInProgress) {
      inputBuffer[bytesRecvd] = x;
      bytesRecvd ++;
      if (bytesRecvd == buffSize) {
        bytesRecvd = buffSize - 1;
      }
    }

if (x == startMarker) {
      bytesRecvd = 0;
      readInProgress = true;
    }
  }
}

//=============

/*

void parseData() {

// split the data into its parts
   
  char * strtokIndx; // this is used by strtok() as an index
 
  strtokIndx = strtok(inputBuffer,",");      // get the first part - the string
  strcpy(messageFromGPS, strtokIndx); // copy it to messageFromPC

}
*/

//=============


The line where I was getting the illegal conversion was 

§DISCOURSE_HOISTED_CODE_1§


and when I couldn't get around this, I changed it to 

§DISCOURSE_HOISTED_CODE_2§


Which I hope will work.

I haven't tried it yet as i am still looking for a way to short the pins so that as soon as the GPS module starts giving out data, the interrupt off pin 2 is activated and wakes the system up.

It has the beginning marker as the dollar sign that each NMEA line begins with, and the ending marker as \n. This means it will only be taking one line per write at the moment which is not what I want, but is a good step forward in regulating the data to be written. I know that the NMEA data has 6 lines, so could easily solve this by changing the main loop to :

§DISCOURSE_HOISTED_CODE_3§


but 6 writes is very inefficient. I'll sort this out after testing this current method works after shorting these pins. Any advice for shorting pins on micro boards?

pickle_64:
I haven't tried it yet as i am still looking for a way to short the pins so that as soon as the GPS module starts giving out data, the interrupt off pin 2 is activated and wakes the system up.

I'm not sure I understand your problem. If you need the same signal to appear on 2 input pins so that one of the pins can react to it by causing an interrupt you can just put some sort of jumper wire between the pins. Just make sure the BOTH pins are NEVER set as OUTPUT at the same time.

Thats exactly what I'm after, it's more about knowing which pins are which on this tiny board