Storing calibration tables on an SD card

Hello,

Well, I have been doing my homework and now I am even more confused. I am not a programmer. I lack the vocabulary to even find an example I can work with. Here's what I am trying to do:

In short, I am trying to have my Arduino read calibration tables from an SD card, store those tables in PROGMEM, and then perform a sort of linear interoperation to find the calibrated value of a temperature sensor. The idea is to be able to swap out SD cards with different calibration tables on them.

My calibration tables are relatively small (I think) at about 31 elements that are signed 2-bit integers. My numbers are typically in this format: 0.235, 1.399, -4.211, etc. I converted them all to integers by multiplying by 100 so I wouldn't have to deal with floats. After retrieving the number from the calibration array, I multiply by 0.01 to get the number back to where it needs to be as a float. I am not sure if this is a waste of effort or not.

I have been loading it into PROGMEM for what seemed to be the fastest way to access the info at runtime.

My array looks like this now:

PROGMEM prog_int16_t balanceArray[] = { 0, 8, -10, 56, 136, 189, 315, 315, 523, 620, 734, 971, 929, 1076, 1343, 1742, 1676, 2110, 2351, 2618, 2796, 2520, 2583, 2556, 2684, 2869, 2788, 2308, 2511, 2529, 2452 };

I want to store the balanceArray data on an SD card. I have a MicroSD shield attached to my Uno. When the Arduino is powered up, I want to retrieve the calibration table from the SD card, and load it into the PROGMEM array for runtime.

My goal is to be able swap SD cards and dynamically load different calibration tables.

Is this possible?

Thank you so much for your help.

Ryan
Boston, MA

Sadly, there is no way for an Arduino sketch to write into progmem. The ATmega328 has 1024 bytes of EEPROM that you can store data into.

You just need to split each 16-bit integer into two 8-bit bytes, store the bytes into EEPROM, then read the bytes back when you need them and merge them into an integer.

http://arduino.cc/en/Reference/LowByte
http://arduino.cc/en/Reference/HighByte

My calibration tables are relatively small (I think) at about 31 elements that are signed 2-bit integers.
That means that the values are in the range 0 to 3. Is that right? Or, did you mean 2 byte integers?

My calibration tables are relatively small (I think) at about 31 elements that are signed 2-bit integers. My numbers are typically in this format: 0.235, 1.399, -4.211, etc. I converted them all to integers by multiplying by 100 so I wouldn't have to deal with floats. After retrieving the number from the calibration array, I multiply by 0.01 to get the number back to where it needs to be as a float. I am not sure if this is a waste of effort or not.

How often do you read/manipulate the array? Once in setup()? Then, yes it's a waste of effort. 25 times a second? Then it may or may not be.

A 31 element array is using only a small part of SRAM. What else is using the rest of SRAM? Or, are you trying to solve a problem that doesn't really exist?

Or to rephrase PaulS: how many of these lookup tables do you have?

My mistake. 2 byte integers. I was using this: prog_uint16_t - an unsigned int (2 bytes) 0 to 65,535

My thinking was that using PROGMEM would allow me to run my device even without an SD card attached. If there was an SD card present, I would retrieve the calibration tables during setup and replace the ones in PROGMEM. If not, run using the latest tables stored in PROGMEM.

But PaulS, I will only be doing the conversion from a calibration value like 1.23 to 123 and back once. Everything after that is just using the lookup table.

I will have about 10+ calibration tables to start, but this will be expanding as this project progresses.

Am I headed in the right direction? Again, I am not new to this type of work... but this is the first time I am doing it on my own. Usually I have my EEs do it and they work their magic. I am trying to learn a bit of magic. I am loving it too...

Thanks for your help.

If there was an SD card present, I would retrieve the calibration tables during setup and replace the ones in PROGMEM.

You can't write to PROGMEM, so this won't work.

Am I headed in the right direction?

I'm thinking no.

If you have an SD card attached, why would you read it to PROGMEM?
A callibration array is 31x2 bytes = 62 bytes so it would fit easily in RAM.

(With an SD card) In your code you should have means to select the name of the callibration file at startup. maybe store the last chosen filename in EEPROM so an unattended reboot can autoload the right file after reboot. Or if no file is selected it automatically loads "default.txt" or so.

Can you tell more about the application what it should do, optionally post the code you have allready?

Let me clean up my code a bit. Right now it is a mishmosh of examples all jammed together.

The project is for a blood pressure monitor I am building. It uses a calibration table from a "reference arm" I am building as well. I want to be able to deploy the monitor which would use the calibration tables on the SD card. As I continue to improve the reference arm, I can take a second card, upload the more accurate calibration table onto the SD card and swap cards with the monitor.

The whole effort with PROGMEM was to speed up the linear interpolation attempts at using the lookup table. I know there isn't a native LERP function on the Arduino, so I have been using examples that I could find. It seemed to be a resource hog, and it sounded like PROGMEM was the way to go. It would be cool if the monitor could store the last used calibration table and continue to operate even if the SD card is taken out. That's fancy though.

I am coming from a LabView RT and FPGA world. I am trying to do something similar with an Arduino now.

For linear interpolation in callibration files you might find my multimap code - Arduino Playground - MultiMap - usefull..

LabviewRT and FPGA are way faster than Arduino, nevertheless an Arduino still can do amazing things ... (some expectation management :slight_smile:

Okay, I am learning.

I am taking small steps toward my bigger goal, but I am stuck. I don’t understand my problem, so I am having a hard time finding the right keywords to search for. Here is where I have gotten with my code:

#include <SdFat.h>
#include <SdFatUtil.h>
#include <string.h>

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

// store error strings in flash to save RAM
#define error(s) error_P(PSTR(s))

char sz[] = "3500,270,890,70,4";

char responseArray[] = {};
    

void error_P(const char* str) {
  PgmPrint("error: ");
  SerialPrintln_P(str);
  if (card.errorCode()) {
    PgmPrint("SD error: ");
    Serial.print(card.errorCode(), HEX);
    Serial.print(',');
    Serial.println(card.errorData(), HEX);
  }
  while(1);
}

void setup(void) {
  Serial.begin(9600);
//  Serial.println();
//  Serial.println("Type any character to start");
//  while (!Serial.available());
  Serial.println();
  
  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!card.init(SPI_HALF_SPEED)) error("card.init failed");
  
  // initialize a FAT volume
  if (!volume.init(&card)) error("volume.init failed");
  
  // open the root directory
  if (!root.openRoot(&volume)) error("openRoot failed");
  
  // open a file
  if (file.open(&root, “SENSOR.TXT", O_READ)) {
    Serial.println("Opened SENSOR.TXT");
  }

  else{
    error("file.open failed");
  }
  Serial.println();
  
  // copy file to serial port
  int16_t n;
  uint8_t buf[7];// nothing special about 7, just a lucky number.
  while ((n = file.read(buf, sizeof(buf))) > 0) {
    for (uint8_t i = 0; i < n; i++) 
    { 
      //Serial.print(buf[i]);
      responseArray[i] = buf[i];
      Serial.print("responseArray[");
      Serial.print(i);
      Serial.print("] = ");
      Serial.println(responseArray[i]);
    }
  }
  /* easier way
  int16_t c;
  while ((c = file.read()) > 0) Serial.print((char)c);
  */
  Serial.println("\nDone");
  
   char *p = sz;
   char *str;

//   while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the semicolon
//   Serial.println(str);
  
}
void loop(void) {}

Here I have two examples combined into one program. Neither piece seems to be working yet, but I feel like I am close. When I run this, I am seeing this:

Opened MDCO.TXT

responseArray[] = 0
responseArray[] = .
responseArray[] = 0
responseArray[] = 0
responseArray[] = 

responseArray[] = 

responseArray[] = 0
responseArray[] = .
responseArray[] = 0
responseArray[] = 8
responseArray[] = 

responseArray[] = 

responseArray[] = -
responseArray[] = 0
responseArray[] = .
responseArray[] = 1
responseArray[] = 0
responseArray[] = 

responseArray[] =

I am FINALLY seeing the data coming in from the SD card, but I don’t seem to be able to put it into the array correctly. First, the i that I am using to increment the array element doesn’t seem to be working correctly. It doesn’t print, so I am assuming that I am putting the value from the TXT file into the same element. No?

Second, how the heck do I combine/append the values that are coming in into the correct format? Right now it looks like I am picking off each individual value and putting that into its own element. My values range from 0.00 to ?0.10 to 12.45 etc. To be honest, as these will be user generated, I can’t guarantee that they will be a known length. What if the user uses something like 12439.122. I would prefer if it wouldn’t break.

I don’t understand enough about bits and bytes and chars to know what the heck I am doing wrong. I understand that I am doing something with each individual character that I retrieve. How do I parse this? I have tried using a CSV format, but couldn’t figure out how to parse it correctly. That seems to be the most straightforward way of doing this. I would simply read from the SD card until I hit a comma, and then insert that value into the array element. No luck though.

This is why I tried using each number on a separate line like this:

(this is what SENSOR.TXT currently looks like)

0.00
0.08
-0.10
0.56
1.36
1.89
3.15
3.15
5.23
6.20
7.34
9.71
9.29
10.76
13.43
17.42
16.76

It seems like my code is even picking up the CR in the text file. Ack, I am a bit lost now. Anyone able to help guide me do something like a CSV parser using my code as a base? I was thrilled to get it this far...

...now I am stumped.

Thanks again for your help.

Ryan

The data is being put into the array correctly. The problem is that you are printing the index, i, as though it was a printable character. It isn't.

Change the type of i from uint8_t to int, and see if that doesn't look better.

This still isn’t working correctly.

I am currently running this code. It reads in a text file in CSV format containing values like 0.01, ?0.12, 23.98, etc. I am trying to put each value into an array. Right now, it seems like I am putting each char into the array. Current code:

#include <SdFat.h>
#include <SdFatUtil.h>
#include <string.h>

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

// store error strings in flash to save RAM
#define error(s) error_P(PSTR(s))



char responseArray[] = {};
    

void error_P(const char* str) {
  PgmPrint("error: ");
  SerialPrintln_P(str);
  if (card.errorCode()) {
    PgmPrint("SD error: ");
    Serial.print(card.errorCode(), HEX);
    Serial.print(',');
    Serial.println(card.errorData(), HEX);
  }
  while(1);
}

void setup(void) {
  Serial.begin(9600);
//  Serial.println();
//  Serial.println("Type any character to start");
//  while (!Serial.available());
  Serial.println();
  
  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!card.init(SPI_HALF_SPEED)) error("card.init failed");
  
  // initialize a FAT volume
  if (!volume.init(&card)) error("volume.init failed");
  
  // open the root directory
  if (!root.openRoot(&volume)) error("openRoot failed");
  
  // open a file
  if (file.open(&root, "CR.TXT", O_READ)) {
    Serial.println("Opened CR.TXT");
  }

  else{
    error("file.open failed");
  }
  Serial.println();
  
  String receivedData;
  
  // copy file to serial port
  int16_t n;
  uint8_t buf[7];// nothing special about 7, just a lucky number.
  while ((n = file.read(buf, sizeof(buf))) > 0) {
    for (int i = 0; i < n; i++) 
    {       
      responseArray[i] = buf[i];  
      Serial.print("responseArray[");
      Serial.print(i);
      Serial.print("] = ");
      Serial.println(responseArray[i]);
    }
  }


  Serial.println("\nDone");
  

  
  Serial.println(responseArray[0]);
  Serial.println(responseArray[1]);
  Serial.println(responseArray[2]);
  Serial.println(responseArray[3]);
  Serial.println(responseArray[4]);
  Serial.println(responseArray[5]);
  Serial.println(responseArray[6]);
}

void loop(void) {

}

My output looks like this:

responseArray[0] = 0
responseArray[1] = .
responseArray[2] = 0
responseArray[3] = 0
responseArray[4] = ,
responseArray[5] =  
responseArray[6] = 0
responseArray[0] = .
responseArray[1] = 0
responseArray[2] = 8
responseArray[3] = ,
responseArray[4] =  
responseArray[5] = -
responseArray[6] = 0

It’s going in there, but it is going in one value at a time.

The last Serial.printlns at the end are telling me that it isn’t working. It seems like I am still missing something in the code that tells it when to stop reading the file, consider those values a single number, and put THAT into the array. I need a CSV parser of some sort. There needs to be something that tells it to stop at a comma, no?

I have found strtok and sscanf ... but no example of how to use them when reading from an SD card. I have done string parsing before, but never when it is coming in nonstop from the SD card. That is where I am lost.

Many thanks.

Ryan

I

responseArray = buf*; [/quote]*
That's why every character comes in one array element.
The problem is in the parsing.
You need to two functions to get things right, something like this one, and the other one is left as an exercise
*_ <em>*float readFloat() {   char buf[10];   readUntil(buf, ',' , 8);  // read from file until 8 chars are read or until a ',' is found   float rv = atof(buf);   return rv; }*</em> _*

responseArray = buf;

Where did you see this? I see

      responseArray[i] = buf[i];

Which is clearly copying stuff into responseArray correctly.

You are correct that the problem is not in the reading of the data. The data is read from the card correctly. There is no reason to copy it to another array.

You can either use robtillaart's approach or use strtok() to extract a token from buf, and pass that token to atof() for conversion to a float.

The size of buf should be larger, though, so that you can read an entire record, not just 7 bytes at a time.

If you go with robtillaart's approach, don't forget that the last value in a record may not be terminated by a ,.

Haha forum bug :slight_smile:

I reread my post but the [ i ] is in the quoted text although without the spaces. It is probably interpreted as an italic(?) flag by the forum engine

Fun

It is probably interpreted as an italic(?) flag by the forum engine

Exactly. b, u, and s in brackets do things, too.

Okay, I am really trying here. This goes WAY beyond my understanding of programming. I have zero C++ background and it seems like strtok() is a native C function. This means the examples I can find are near impossible to understand. I have managed to get it to do what seems right (after a ton of trial and error) but it isn’t right.

Here is what I understand my problems to be:

  1. char buf[6]; is a problem. I don’t necessarily know what the length of the calibration value is. Some are like 0.00, some are like ?0.10, and most are like 12.34. I tried char buf[5] but that starting returning gibberish. Basically, I don’t want this at all. I want to retrieve just the chunks in between the commas. I can’t get that to work.

  2. My debug output looks great... but it isn’t inserting the right values into the array. It seems like it is putting the last value into every array element.

  3. Right now, I am swapping variable types just to get the thing to compile. I want to build an array of floats. To get this to work, I can only get chars in there. I don’t seem to be able to parse the serial data for a comma without using chars.

#include <SdFat.h>
#include <SdFatUtil.h>

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

char* responseArray[] = {};

// store error strings in flash to save RAM
#define error(s) error_P(PSTR(s))

void error_P(const char* str) {
  PgmPrint("error: ");
  SerialPrintln_P(str);
  if (card.errorCode()) {
    PgmPrint("SD error: ");
    Serial.print(card.errorCode(), HEX);
    Serial.print(',');
    Serial.println(card.errorData(), HEX);
  }
  while(1);
}

void setup(void) {
  
  Serial.begin(9600);
  Serial.println();
  Serial.println("Type any character to start");
  while (!Serial.available());
  Serial.println();
  
  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!card.init(SPI_HALF_SPEED)) error("card.init failed");
  
  // initialize a FAT volume
  if (!volume.init(&card)) error("volume.init failed");
  
  // open the root directory
  if (!root.openRoot(&volume)) error("openRoot failed");
  
  // open a file
  if (file.open(&root, "CSV.TXT", O_READ)) {
    Serial.println("Opened CSV.TXT");
  }
  
  else{
    error("file.open failed");
  }
  
  int16_t n;
  char* parameter;  
  int parserIteration = 0;
  
  char buf[6]; // THIS IS BASICALLY HARD-CODING. I CAN'T GUARANTEE I KNOW THE LENGTH OF THE VALUES IN THE TXT FILE. 
  
  while ((n = file.read(buf, sizeof(buf))) > 0) 
  {
      Serial.print("Entering ");
      parameter = strtok(buf, ",");        
      Serial.print(parameter);
      Serial.print(" into responseArray[");
      Serial.print(parserIteration);
      Serial.println("]");
      
      responseArray[parserIteration] = parameter; // THESE VALUES ARE GOING INTO THE ARRAY AS CHAR* ... I NEED THEM IN THERE AS FLOATS.
      
      parserIteration++;
        
      while (parameter != NULL)
      {
        parameter = strtok(NULL, ","); // NOT SURE WHAT THIS DOES, BUT IT BREAKS IF I DON'T HAVE IT
      }
  }

  Serial.println("\nDone\n");
  
  Serial.println("Now, test the array...\n");
  
  // AND THIS DOESN'T REALLY WORK AT ALL, DESPITE LOOKING CORRECT IN MY DEBUG OUTPUT
  
  Serial.print("responseArray[0] now = ");
  Serial.println(responseArray[0]);
  Serial.print("responseArray[1] now = ");
  Serial.println(responseArray[1]);
  Serial.print("responseArray[18] now = ");
  Serial.println(responseArray[18]);
}

void loop(void) {}

and my seemingly awesome serial output that does nothing...

Type any character to start

Opened CSV.TXT
Entering 00.00 into responseArray[0]
Entering 00.08 into responseArray[1]
Entering -0.10 into responseArray[2]
Entering 00.56 into responseArray[3]
Entering 01.36 into responseArray[4]
Entering 01.89 into responseArray[5]
Entering 03.15 into responseArray[6]
Entering 03.15 into responseArray[7]
Entering 05.23 into responseArray[8]
Entering 06.20 into responseArray[9]
Entering 07.34 into responseArray[10]
Entering 09.71 into responseArray[11]
Entering 09.29 into responseArray[12]
Entering 10.76 into responseArray[13]
Entering 13.43 into responseArray[14]
Entering 17.42 into responseArray[15]
Entering 16.76 into responseArray[16]
Entering 21.10 into responseArray[17]
Entering 23.51 into responseArray[18]
Entering 26.18 into responseArray[19]
Entering 27.96 into responseArray[20]
Entering 25.20 into responseArray[21]
Entering 25.83 into responseArray[22]
Entering 25.56 into responseArray[23]
Entering 26.84 into responseArray[24]
Entering 28.69 into responseArray[25]
Entering 27.88 into responseArray[26]
Entering 23.08 into responseArray[27]
Entering 25.11 into responseArray[28]
Entering 25.29 into responseArray[29]
Entering 24.52
*@ into responseArray[30]

Done

Now, test the array...

responseArray[0] now = 24.52
*@
responseArray[1] now = 
responseArray[18] now = 24.52
*@

Getting closer!

I liked this approach Rob, particularly because you said, “read from a file until 8 chars are read OR a “,” is found” That OR is exactly what I want. My problem is I can’t figure out the syntax of what should be in a readUntil() function.

robtillaart:

float readFloat()

{
  char buf[10];
  readUntil(buf, ',' , 8);  // read from file until 8 chars are read or until a ',' is found
  float rv = atof(buf);
  return rv;
}

PaulS:
You are correct that the problem is not in the reading of the data. The data is read from the card correctly. There is no reason to copy it to another array.

Hi Paul, in rereading your comments, I am realizing that I may be missing the big picture. I thought I had to put the data into an array so I could access it later on. My idea was to read the calibration table from the SD card during setup, and then use it with some form of linear interpolation during the main loop. I only want to read the SD card once, no?

Am I missing a best practice for this type of stuff? Learning a lot, that’s for sure.

Ryan

I thought I had to put the data into an array so I could access it later on.

You do. But that is exactly what file.read() is doing. You do not need to copy that array to another array.

What you want to store in another array is the numbers read from the file, not the characters.

You could create a readUntil() function that took an array of delimiters (or a hardcoded set). Make that function read one character at a time from the file. Compare that character to each value in the array of delimiters. If no match, store the character in the output array. If there is a match, return.

You would then call that function with a comma, a carriage return, and a line feed in the array of delimiters. The function would return a string that contained the representation of a single floating point number, which atof() would like.

Be sure to check the length of the string before calling atof(). The carriage return and line feed are adjacent in the file, so with both as delimiters, the function would return an empty string after the first was found, when the second one was. You want to ignore that empty string.