MicroSD Weirdness

Using Arduino Storage
1

Hey Everyone,

I'm fairly new to the whole arduino world, and this is my first venture into datalogging territory, and of course, I'm running up against a wall. My goal is to read data of an IR array (MLX90620) and then save the temperatures to an SD card, exactly how it's printing to the Serial Monitor.

I've run through several SD library examples, and they seem to work - I can log the sketch data on the SD card, and then verify by popping it into my computer. All good. I can get temperatures from the MLX90620. All good. But I can't seem to get them to work together! This is the sketch that reads the MLX90620 - I have to break it up to fit here, so here's the first half:

/*
 2-16-2013
 Spark Fun Electronics
 Nathan Seidle
 
 This code is heavily based on maxbot's and IlBaboomba's code: http://arduino.cc/forum/index.php?topic=126244
 They didn't have a license on it so I'm hoping it's public domain.
 
 This example shows how to read and calculate the 64 temperatures for the 64 pixels of the MLX90620 thermopile sensor.
 
 alpha_ij array is specific to every sensor and needs to be calculated separately. Please see the 
 'MLX90620_alphaCalculator' sketch to get these values. If you choose not to calculate these values
 this sketch will still work but the temperatures shown will be very inaccurate.
 
 Don't get confused by the bottom view of the device! The GND pin is connected to the housing.
 
 To get this code to work, attached a MLX90620 to an Arduino Uno using the following pins:
 A5 to 330 ohm to SCL
 A4 to 330 ohm to SDA
 3.3V to VDD
 GND to VSS
 
 I used the internal pull-ups on the SDA/SCL lines. Normally you should use ~4.7k pull-ups for I2C.

 */

#include <i2cmaster.h>
//i2cmaster comes from here: http://www.cheap-thermocam.bplaced.net/software/I2Cmaster.rar

#include "MLX90620_registers.h"

int refreshRate = 16; //Set this value to your desired refresh frequency


//Global variables
//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
int irData[64]; //Contains the raw IR data from the sensor
float temperatures[64]; //Contains the calculated temperatures of each pixel in the array
float Tambient; //Tracks the changing ambient temperature of the sensor
byte eepromData[256]; //Contains the full EEPROM reading from the MLX (Slave 0x50)

//These are constants calculated from the calibration data stored in EEPROM
//See varInitialize and section 7.3 for more information
int v_th, a_cp, b_cp, tgc, b_i_scale;
float k_t1, k_t2, emissivity;
int a_ij[64], b_ij[64];

//These values are calculated using equation 7.3.3.2
//They are constants and can be calculated using the MLX90620_alphaCalculator sketch

float alpha_ij[64] = {

  1.46447E-8, 1.50521E-8, 1.50521E-8, 1.34805E-8, 1.60126E-8, 1.62163E-8, 1.65946E-8, 1.48484E-8, 

  1.62163E-8, 1.65946E-8, 1.64200E-8, 1.50521E-8, 1.65946E-8, 1.73805E-8, 1.72058E-8, 1.60126E-8, 

  1.73805E-8, 1.77879E-8, 1.79625E-8, 1.62163E-8, 1.83700E-8, 1.85446E-8, 1.83700E-8, 1.67984E-8, 

  1.87483E-8, 1.93304E-8, 1.89521E-8, 1.70021E-8, 1.89521E-8, 1.97379E-8, 1.91558E-8, 1.77879E-8, 

  1.93304E-8, 1.93304E-8, 1.93304E-8, 1.79625E-8, 1.89521E-8, 1.95341E-8, 1.95341E-8, 1.79625E-8, 

  1.89521E-8, 1.95341E-8, 1.97379E-8, 1.79625E-8, 1.87483E-8, 1.89521E-8, 1.91558E-8, 1.73805E-8, 

  1.81663E-8, 1.85737E-8, 1.87483E-8, 1.70021E-8, 1.72058E-8, 1.81663E-8, 1.79625E-8, 1.72058E-8, 

  1.65946E-8, 1.75842E-8, 1.79625E-8, 1.65946E-8, 1.62163E-8, 1.72058E-8, 1.72058E-8, 1.58379E-8, 

};
byte loopCount = 0; //Used in main loop
//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=


//Begin Program code

void setup()
{
  Serial.begin(115200);
  Serial.println("MLX90620 Example");

  i2c_init(); //Init the I2C pins
  PORTC = (1 << PORTC4) | (1 << PORTC5); //Enable pull-ups

  delay(5); //Init procedure calls for a 5ms delay after power-on

  read_EEPROM_MLX90620(); //Read the entire EEPROM

  setConfiguration(refreshRate); //Configure the MLX sensor with the user's choice of refresh rate

  calculate_TA(); //Calculate the current Tambient
}

void loop()
{
  if(loopCount++ == 16) //Tambient changes more slowly than the pixel readings. Update TA only every 16 loops.
  { 
    calculate_TA(); //Calculate the new Tambient

    if(checkConfig_MLX90620()) //Every 16 readings check that the POR flag is not set
    {
      Serial.println("POR Detected!");
      setConfiguration(refreshRate); //Re-write the configuration bytes to the MLX
    }

    loopCount = 0; //Reset count
  }

  readIR_MLX90620(); //Get the 64 bytes of raw pixel data into the irData array

  calculate_TO(); //Run all the large calculations to get the temperature data for each pixel

  prettyPrintTemperatures(); //Print the array in a 4 x 16 pattern
  //rawPrintTemperatures(); //Print the entire array so it can more easily be read by Processing app
}

//From the 256 bytes of EEPROM data, initialize 
void varInitialization(byte calibration_data[])
{
  v_th = 256 * calibration_data[VTH_H] + calibration_data[VTH_L];
  k_t1 = (256 * calibration_data[KT1_H] + calibration_data[KT1_L]) / 1024.0; //2^10 = 1024
  k_t2 = (256 * calibration_data[KT2_H] + calibration_data[KT2_L]) / 1048576.0; //2^20 = 1,048,576
  emissivity = ((unsigned int)256 * calibration_data[CAL_EMIS_H] + calibration_data[CAL_EMIS_L]) / 32768.0;
  
  a_cp = calibration_data[CAL_ACP];
  if(a_cp > 127) a_cp -= 256; //These values are stored as 2's compliment. This coverts it if necessary.

  b_cp = calibration_data[CAL_BCP];
  if(b_cp > 127) b_cp -= 256;

  tgc = calibration_data[CAL_TGC];
  if(tgc > 127) tgc -= 256;

  b_i_scale = calibration_data[CAL_BI_SCALE];

  for(int i = 0 ; i < 64 ; i++)
  {
    //Read the individual pixel offsets
    a_ij[i] = calibration_data[i]; 
    if(a_ij[i] > 127) a_ij[i] -= 256; //These values are stored as 2's compliment. This coverts it if necessary.

    //Read the individual pixel offset slope coefficients
    b_ij[i] = calibration_data[0x40 + i]; //Bi(i,j) begins 64 bytes into EEPROM at 0x40
    if(b_ij[i] > 127) b_ij[i] -= 256;
  }
  
}

//Receives the refresh rate for sensor scanning
//Sets the two byte configuration registers
//This function overwrites what is currently in the configuration registers
//The MLX doesn't seem to mind this (flags are read only)
void setConfiguration(int irRefreshRateHZ)
{
  byte Hz_LSB;

  switch(irRefreshRateHZ)
  {
  case 0:
    Hz_LSB = 0b00001111;
    break;
  case 1:
    Hz_LSB = 0b00001110;
    break;
  case 2:
    Hz_LSB = 0b00001101;
    break;
  case 4:
    Hz_LSB = 0b00001100;
    break;
  case 8:
    Hz_LSB = 0b00001011;
    break;
  case 16:
    Hz_LSB = 0b00001010;
    break;
  case 32:
    Hz_LSB = 0b00001001;
    break;
  default:
    Hz_LSB = 0b00001110;
  }

  byte defaultConfig_H = 0b01110100; // x111.01xx, Assumes NA = 0, ADC low reference enabled, Ta Refresh rate of 2Hz

  i2c_start_wait(MLX90620_WRITE);
  i2c_write(0x03); //Command = configuration value
  i2c_write((byte)Hz_LSB - 0x55);
  i2c_write(Hz_LSB);
  i2c_write(defaultConfig_H - 0x55); //Assumes NA = 0, ADC low reference enabled, Ta Refresh rate of 2Hz
  i2c_write(defaultConfig_H);
  i2c_stop();
}

//Read the 256 bytes from the MLX EEPROM and setup the various constants (*lots* of math)
//Note: The EEPROM on the MLX has a different I2C address from the MLX. I've never seen this before.
void read_EEPROM_MLX90620()
{
  i2c_start_wait(MLX90620_EEPROM_WRITE);
  i2c_write(0x00); //EEPROM info starts at location 0x00
  i2c_rep_start(MLX90620_EEPROM_READ);

  //Read all 256 bytes from the sensor's EEPROM
  for(int i = 0 ; i <= 255 ; i++)
    eepromData[i] = i2c_readAck();

  i2c_stop(); //We're done talking

  varInitialization(eepromData); //Calculate a bunch of constants from the EEPROM data

  writeTrimmingValue(eepromData[OSC_TRIM_VALUE]);
}
2

and the second half:

//Given a 8-bit number from EEPROM (Slave address 0x50), write value to MLX sensor (Slave address 0x60)
void writeTrimmingValue(byte val)
{
  i2c_start_wait(MLX90620_WRITE); //Write to the sensor
  i2c_write(0x04); //Command = write oscillator trimming value
  i2c_write((byte)val - 0xAA);
  i2c_write(val);
  i2c_write(0x56); //Always 0x56
  i2c_write(0x00); //Always 0x00
  i2c_stop();
}

//Gets the latest PTAT (package temperature ambient) reading from the MLX
//Then calculates a new Tambient
//Many of these values (k_t1, v_th, etc) come from varInitialization and EEPROM reading
//This has been tested to match example 7.3.2
void calculate_TA(void)
{
  unsigned int ptat = readPTAT_MLX90620();

  Tambient = (-k_t1 + sqrt(square(k_t1) - (4 * k_t2 * (v_th - (float)ptat)))) / (2*k_t2) + 25; //it's much more simple now, isn't it? :)
}

//Reads the PTAT data from the MLX
//Returns an unsigned int containing the PTAT
unsigned int readPTAT_MLX90620()
{
  i2c_start_wait(MLX90620_WRITE);
  i2c_write(CMD_READ_REGISTER); //Command = read PTAT
  i2c_write(0x90); //Start address is 0x90
  i2c_write(0x00); //Address step is 0
  i2c_write(0x01); //Number of reads is 1
  i2c_rep_start(MLX90620_READ);

  byte ptatLow = i2c_readAck(); //Grab the lower and higher PTAT bytes
  byte ptatHigh = i2c_readAck();

  i2c_stop();
  
  return( (unsigned int)(ptatHigh << 8) | ptatLow); //Combine bytes and return
}

//Calculate the temperatures seen for each pixel
//Relies on the raw irData array
//Returns an 64-int array called temperatures
void calculate_TO()
{
  float v_ir_off_comp;
  float v_ir_tgc_comp;
  float v_ir_comp;

  //Calculate the offset compensation for the one compensation pixel
  //This is a constant in the TO calculation, so calculate it here.
  int cpix = readCPIX_MLX90620(); //Go get the raw data of the compensation pixel
  float v_cp_off_comp = (float)cpix - (a_cp + (b_cp/pow(2, b_i_scale)) * (Tambient - 25)); 

  for (int i = 0 ; i < 64 ; i++)
  {
    v_ir_off_comp = irData[i] - (a_ij[i] + (float)(b_ij[i]/pow(2, b_i_scale)) * (Tambient - 25)); //#1: Calculate Offset Compensation 

    v_ir_tgc_comp = v_ir_off_comp - ( ((float)tgc/32) * v_cp_off_comp); //#2: Calculate Thermal Gradien Compensation (TGC)

    v_ir_comp = v_ir_tgc_comp / emissivity; //#3: Calculate Emissivity Compensation

    temperatures[i] = sqrt( sqrt( (v_ir_comp/alpha_ij[i]) + pow(Tambient + 273.15, 4) )) - 273.15;
  }
}

//Reads 64 bytes of pixel data from the MLX
//Loads the data into the irData array
void readIR_MLX90620()
{
  i2c_start_wait(MLX90620_WRITE);
  i2c_write(CMD_READ_REGISTER); //Command = read a register
  i2c_write(0x00); //Start address = 0x00
  i2c_write(0x01); //Address step = 1
  i2c_write(0x40); //Number of reads is 64
  i2c_rep_start(MLX90620_READ);

  for(int i = 0 ; i < 64 ; i++)
  {
    byte pixelDataLow = i2c_readAck();
    byte pixelDataHigh = i2c_readAck();
    irData[i] = (int)(pixelDataHigh << 8) | pixelDataLow;
  }

  i2c_stop();
}

//Read the compensation pixel 16 bit data
int readCPIX_MLX90620()
{
  i2c_start_wait(MLX90620_WRITE);
  i2c_write(CMD_READ_REGISTER); //Command = read register
  i2c_write(0x91);
  i2c_write(0x00);
  i2c_write(0x01);
  i2c_rep_start(MLX90620_READ);

  byte cpixLow = i2c_readAck(); //Grab the two bytes
  byte cpixHigh = i2c_readAck();
  i2c_stop();

  return ( (int)(cpixHigh << 8) | cpixLow);
}

//Reads the current configuration register (2 bytes) from the MLX
//Returns two bytes
unsigned int readConfig_MLX90620()
{
  i2c_start_wait(MLX90620_WRITE); //The MLX configuration is in the MLX, not EEPROM
  i2c_write(CMD_READ_REGISTER); //Command = read configuration register
  i2c_write(0x92); //Start address
  i2c_write(0x00); //Address step of zero
  i2c_write(0x01); //Number of reads is 1

    i2c_rep_start(MLX90620_READ);

  byte configLow = i2c_readAck(); //Grab the two bytes
  byte configHigh = i2c_readAck();

  i2c_stop();

  return( (unsigned int)(configHigh << 8) | configLow); //Combine the configuration bytes and return as one unsigned int
}

//Poll the MLX for its current status
//Returns true if the POR/Brown out bit is set
boolean checkConfig_MLX90620()
{
  if ( (readConfig_MLX90620() & (unsigned int)1<<POR_TEST) == 0)
    return true;
  else
    return false;
}

//Prints the temperatures in a way that's more easily viewable in the terminal window
void prettyPrintTemperatures()
{
  Serial.println();
  for(int i = 0 ; i < 64 ; i++)
  {
    if(i % 16 == 0) Serial.println();
    Serial.print(temperatures[i]);
    //Serial.print(irData[i]);
    Serial.print("\t");
  }
}

//Prints the temperatures in a way that's more easily parsed by a Processing app
//Each line starts with '

The weird thing is that the code spits out data fine to the serial until I add the SD library. If I upload the file with no changes other than just adding the library, I get no data over serial whatsoever. Anyone know why these two aren't playing well together, and how I can get them working?

If I matters, I'm running a lilypad bootloader on a atmega328...

Thanks for any suggestions!

Cheers,
P and ends with ''
void rawPrintTemperatures()
{
  Serial.print("$");
  for(int i = 0 ; i < 64 ; i++)
  {
    Serial.print(temperatures[i]);
    Serial.print(","); //Don't print comma on last temperature
  }
  Serial.println("");
}

//Given a Celsius float, converts to Fahrenheit
float convertToFahrenheit (float Tc)
{
  float Tf = (9/5) * Tc + 32;

return(Tf);
}


The weird thing is that the code spits out data fine to the serial until I add the SD library. If I upload the file with no changes other than just adding the library, I get no data over serial whatsoever. Anyone know why these two aren't playing well together, and how I can get them working? 

If I matters, I'm running a lilypad bootloader on a atmega328...

Thanks for any suggestions!

Cheers,
P
3

You have used most of the 328's RAM on arrays like this:

int irData[64]; //Contains the raw IR data from the sensor
float temperatures[64]; //Contains the calculated temperatures of each pixel in the array
byte eepromData[256]; //Contains the full EEPROM reading from the MLX (Slave 0x50)
int a_ij[64], b_ij[64];
float alpha_ij[64] = {

The SD library requires a 512 byte block buffer plus about 100 additional bytes for file system structures so your program will fail due to lack of RAM.

4

Hey fatlib16,

Thanks for the reply and all the work you put into the SD library - I'm lost now, but I'd be totally in the weeds without the work you've done and shared with others, so thanks for that!

As far as my current memory problem - is there any workarounds for that? I really don't want to alter the program that much as it needs all those arrays. I'm going to add a gps to the whole thing too, so it's going to get bigger!! Is there another library that wouldn't need as much RAM, or should I just switch over to a Mega?

Cheers,

P

5

SD cards are block devices so it is necessary to read and write entire 512 byte blocks. Only very limited write access to an SD is possible with out a 512 byte block buffer.

You are close to having memory problems before adding an SD. You will need a board with more RAM unless you can eliminate some arrays.

6

Hi fat16lib,

Thanks a ton for the heads up; I'll switch everything over to my Mega and give it a go!

Thanks so much for your time!

Cheers,
P

7

Hola!

Looks like the advice you gave was spot on - it works like a champ on a MEGA! Thank you so much - it made my week!! :slight_smile:

One more quick question that someone might be able to shine some light on. This will be a data-logging project running in the field off of a lipo and I want it to run for the longest time possible. If I save the data to a string and then write to the SD every 64 temperatures (that's how many are in the IR array), would I be gaining any power savings? My gut tells me yes, since I don't be opening/writing/closing the SD card for every temp like I am with this code, but I just wanted to double check with those in the know.

void prettyPrintTemperatures()
{ File dataFile = SD.open("data.txt", FILE_WRITE);
  if (dataFile){
  Serial.println();
  dataFile.println();
  for(int i = 0 ; i < 64 ; i++)
  {
    if(i % 16 == 0) Serial.println();
    if(i % 16 == 0) dataFile.println();
    Serial.print(temperatures[i]);
    dataFile.print(temperatures[i]);
    //Serial.print(irData[i]);
    Serial.print("\t");
    dataFile.print("\t");
}
dataFile.close();
}
}

Also, any idea if this string technique would speed up the readings from the IR array? I'd like to take as many data points as I possibly can. Is the process of writing to an SD a blocking function? If so, I would have to consider something like saving to a string anyways since I'd like a temperature "snapshot" at a each gps reading I get. The goal of my final project is to monitor ground temperatures, and the IR sensor, sd breakoutboard and gps will mounted inside an RC plane.

Thanks for any and all advice!
Cheers,
P

8

Opening and closing the file will take a long time and use a lot of power. Also SD.h may not allow the SD to go into low power sleep mode. SD.h is based on a very old version of SdFat and has some problems for low power use.

High power use only occurs while a 512 byte block is written to the SD. If you open a file once and only call flush() when you want to insure data is written to the SD, you will save power.

There are a number of posts in this forum that measure SD power use.

9

Thanks fat16lib,

That clears things up - I thought it was writing to the SD after every "dataFile.print" command. After researching a little I'm under the impression that the data is only written to the SD when the dataFile.close() command is called, which forces a flush of the data. Is that correct?

I've also read that the SD library has an automatic flush when it reaches 512 bytes, but it shouldn't be relied on because it may fail occasionally, resulting in lost data. So here's my solution: run through the prettyPrintTemperatures() function x many times until I'm close to hitting that 512 byte limit, and then run a dataFile.close()? Does that make sense? Or do I run the risk of loosing data that way too?

I've tried to figure out how many iterations to run prettyPrintTemperatures() to get near that 512 mark, but I'm a little confused as to what constitutes a bit. For example, would this- 23.20 - constitute 5 bits? What about the spaces I've inserted with dataFile.write("/t") between readings - do they count for anything?

10

I've also read that the SD library has an automatic flush when it reaches 512 bytes, but it shouldn't be relied on because it may fail occasionally, resulting in lost data.

No the "automatic write" does does not fail occasionally. You will not lose data if you close the file when you are done.

Do not close and reopen the file. A call to open can be very costly since a linear search of the entire directory may be required. A call to flush() is the same as a close followed by an open but minimizes SD accesses. You only need to call flush() if you fear your program will crash or lose power before you close the file. Data written after the flush call will be lost if the file is not closed.

Close and flush do more than writing the block cache, these calls also update the directory entry which is necessary since the file size is stored in the directory entry.

Counting 512 byte intervals is of no value in saving data, all data written before a flush/close call is saved. If you don't close a file, all data written after the last open or flush is lost.

11

Hey fat16lib,

Just to clarify then - if I change my code from dataFile.close() to dataFile.flush() I'll be able to write to SD faster since it doesn't open and close the file?

You only need to call flush() if you fear your program will crash or lose power before you close the file.

What if I'm willing to bet that the program won't crash? Do I still have to put in a dataFile.close() in there to save data, or does the program automatically do that? Or, do I need to flush, and then close?!?

In this older post, you recommend not using flush(), so I'm even more confused now...

The fastest way to log data is to never call flush but make sure the file is closed. flush and close update the file's directory entry. Data will be automatically written when the buffer is full. The directory entry for the file is only updated by flush/close.

flush is a poor name for the operation, that is why it is often call sync. It does more than flush data to the SD, it synchronizes file structures on the SD with cached values in memory.

For fastest data logging I use a button to start and stop the logger. The first time the button is pressed I start logging and light an LED. I just do writes/prints of the data with no flush calls. The second time the button is pressed I stop logging, close the file, and turn off the LED. I never call flush/sync.

12

In this older post, you recommend not using flush(), so I'm even more confused now...

I only suggested calling flush instead of close/open which you proposed. People sometimes call flush after each data record if they fear a crash or power failure. These flush calls are slow but not nearly as slow as a close followed by an open.

The fastest way to write a file is to open the file once in setup, write data to the file with any number of print and write statements, and then close the file when you are done logging data.

What if I'm willing to bet that the program won't crash? Do I still have to put in a dataFile.close() in there to save data, or does the program automatically do that?

You must call close since there is no way to know when to automatically close a file on Arduino.

Once again, all data in the file will be lost if you do not close the file. The close call updates the directory entry for the file to indicate which clusters contain data and the total byte count in the file.

If you do not close the file the file data will be in "lost clusters". A scan of the file system on a PC will recover these clusters but will not repair the file which was not closed.