Storing calibration tables on an SD card

[81Pantah]

Hello,

Well, I have been doing my homework and now I am even more confused. I am not a programmer. I lack the vocabulary to even find an example I can work with. Here's what I am trying to do:

In short, I am trying to have my Arduino read calibration tables from an SD card, store those tables in PROGMEM, and then perform a sort of linear interoperation to find the calibrated value of a temperature sensor. The idea is to be able to swap out SD cards with different calibration tables on them.

My calibration tables are relatively small (I think) at about 31 elements that are signed 2-bit integers. My numbers are typically in this format: 0.235, 1.399, -4.211, etc. I converted them all to integers by multiplying by 100 so I wouldn't have to deal with floats. After retrieving the number from the calibration array, I multiply by 0.01 to get the number back to where it needs to be as a float. I am not sure if this is a waste of effort or not.

I have been loading it into PROGMEM for what seemed to be the fastest way to access the info at runtime.

My array looks like this now:

PROGMEM prog_int16_t balanceArray[] = { 0, 8, -10, 56, 136, 189, 315, 315, 523, 620, 734, 971, 929, 1076, 1343, 1742, 1676, 2110, 2351, 2618, 2796, 2520, 2583, 2556, 2684, 2869, 2788, 2308, 2511, 2529, 2452 };
   
I want to store the balanceArray data on an SD card. I have a MicroSD shield attached to my Uno. When the Arduino is powered up, I want to retrieve the calibration table from the SD card, and load it into the PROGMEM array for runtime.

My goal is to be able swap SD cards and dynamically load different calibration tables.

Is this possible?

Thank you so much for your help.

Ryan
Boston, MA

[johnwasser]

Sadly, there is no way for an Arduino sketch to write into progmem.  The ATmega328 has 1024 bytes of EEPROM that you can store data into.

http://www.arduino.cc/en/Reference/EEPROM

You just need to split each 16-bit integer into two 8-bit bytes, store the bytes into EEPROM, then read the bytes back when you need them and merge them into an integer.

http://arduino.cc/en/Reference/LowByte
http://arduino.cc/en/Reference/HighByte
http://arduino.cc/en/Reference/WordCast

[PaulS]

My calibration tables are relatively small (I think) at about 31 elements that are signed 2-bit integers.
That means that the values are in the range 0 to 3. Is that right? Or, did you mean 2 byte integers?

My calibration tables are relatively small (I think) at about 31 elements that are signed 2-bit integers. My numbers are typically in this format: 0.235, 1.399, -4.211, etc. I converted them all to integers by multiplying by 100 so I wouldn't have to deal with floats. After retrieving the number from the calibration array, I multiply by 0.01 to get the number back to where it needs to be as a float. I am not sure if this is a waste of effort or not.

How often do you read/manipulate the array? Once in setup()? Then, yes it's a waste of effort. 25 times a second? Then it may or may not be.

A 31 element array is using only a small part of SRAM. What else is using the rest of SRAM? Or, are you trying to solve a problem that doesn't really exist?

[robtillaart]

Or to rephrase PaulS: how many of these lookup tables do you have?

[81Pantah]

My mistake. 2 byte integers. I was using this: prog_uint16_t  - an unsigned int (2 bytes) 0 to 65,535

My thinking was that using PROGMEM would allow me to run my device even without an SD card attached. If there was an SD card present, I would retrieve the calibration tables during setup and replace the ones in PROGMEM. If not, run using the latest tables stored in PROGMEM.

But PaulS, I will only be doing the conversion from a calibration value like 1.23 to 123 and back once. Everything after that is just using the lookup table.

I will have about 10+ calibration tables to start, but this will be expanding as this project progresses.

Am I headed in the right direction? Again, I am not new to this type of work... but this is the first time I am doing it on my own. Usually I have my EEs do it and they work their magic. I am trying to learn a bit of magic. I am loving it too...

Thanks for your help.

[PaulS]

If there was an SD card present, I would retrieve the calibration tables during setup and replace the ones in PROGMEM.

You can't write to PROGMEM, so this won't work.

Am I headed in the right direction?

I'm thinking no.

[robtillaart]

If you have an SD card attached, why would you read it to PROGMEM?
A callibration array is 31x2 bytes = 62 bytes so it would fit easily in RAM.

(With an SD card) In your code you should have means to select the name of the callibration file at startup. maybe store the last chosen filename in EEPROM so an unattended reboot can autoload the right file after reboot. Or if no file is selected it automatically loads "default.txt" or so.

Can you tell more about the application what it should do, optionally post the code you have allready?

[81Pantah]

Let me clean up my code a bit. Right now it is a mishmosh of examples all jammed together.

The project is for a blood pressure monitor I am building. It uses a calibration table from a "reference arm" I am building as well. I want to be able to deploy the monitor which would use the calibration tables on the SD card. As I continue to improve the reference arm, I can take a second card, upload the more accurate calibration table onto the SD card and swap cards with the monitor.

The whole effort with PROGMEM was to speed up the linear interpolation attempts at using the lookup table. I know there isn't a native LERP function on the Arduino, so I have been using examples that I could find. It seemed to be a resource hog, and it sounded like PROGMEM was the way to go. It would be cool if the monitor could store the last used calibration table and continue to operate even if the SD card is taken out. That's fancy though.

I am coming from a LabView RT and FPGA world. I am trying to do something similar with an Arduino now.

[robtillaart]

For linear interpolation in callibration files you might find my multimap code - http://arduino.cc/playground/Main/MultiMap - usefull..

LabviewRT and FPGA are way faster than Arduino, nevertheless an Arduino still can do amazing things ... (some expectation management

[81Pantah]

Okay, I am learning.

I am taking small steps toward my bigger goal, but I am stuck. I don't understand my problem, so I am having a hard time finding the right keywords to search for. Here is where I have gotten with my code:

Code: [Select]



#include <SdFat.h>
#include <SdFatUtil.h>
#include <string.h>

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

// store error strings in flash to save RAM
#define error(s) error_P(PSTR(s))

char sz[] = "3500,270,890,70,4";

char responseArray[] = {};
   

void error_P(const char* str) {
  PgmPrint("error: ");
  SerialPrintln_P(str);
  if (card.errorCode()) {
    PgmPrint("SD error: ");
    Serial.print(card.errorCode(), HEX);
    Serial.print(',');
    Serial.println(card.errorData(), HEX);
  }
  while(1);
}

void setup(void) {
  Serial.begin(9600);
//  Serial.println();
//  Serial.println("Type any character to start");
//  while (!Serial.available());
  Serial.println();
 
  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!card.init(SPI_HALF_SPEED)) error("card.init failed");
 
  // initialize a FAT volume
  if (!volume.init(&card)) error("volume.init failed");
 
  // open the root directory
  if (!root.openRoot(&volume)) error("openRoot failed");
 
  // open a file
  if (file.open(&root, "SENSOR.TXT", O_READ)) {
    Serial.println("Opened SENSOR.TXT");
  }

  else{
    error("file.open failed");
  }
  Serial.println();
 
  // copy file to serial port
  int16_t n;
  uint8_t buf[7];// nothing special about 7, just a lucky number.
  while ((n = file.read(buf, sizeof(buf))) > 0) {
    for (uint8_t i = 0; i < n; i++)
    {
      //Serial.print(buf[i]);
      responseArray[i] = buf[i];
      Serial.print("responseArray[");
      Serial.print(i);
      Serial.print("] = ");
      Serial.println(responseArray[i]);
    }
  }
  /* easier way
  int16_t c;
  while ((c = file.read()) > 0) Serial.print((char)c);
  */
  Serial.println("\nDone");
 
   char *p = sz;
   char *str;

//   while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the semicolon
//   Serial.println(str);
 
}
void loop(void) {}


Here I have two examples combined into one program. Neither piece seems to be working yet, but I feel like I am close. When I run this, I am seeing this:

Code: [Select]

Opened MDCO.TXT

responseArray[] = 0
responseArray[
] = .
responseArray[] = 0
responseArray[] = 0
responseArray[] =

responseArray[] =

responseArray[] = 0
responseArray[] = .
responseArray[
] = 0
responseArray[] = 8
responseArray[] =

responseArray[] =

responseArray[] = -
responseArray[] = 0
responseArray[] = .
responseArray[
] = 1
responseArray[] = 0
responseArray[] =

responseArray[] =


I am FINALLY seeing the data coming in from the SD card, but I don't seem to be able to put it into the array correctly. First, the i that I am using to increment the array element doesn't seem to be working correctly. It doesn't print, so I am assuming that I am putting the value from the TXT file into the same element. No?

Second, how the heck do I combine/append the values that are coming in into the correct format? Right now it looks like I am picking off each individual value and putting that into its own element. My values range from 0.00 to ?0.10 to 12.45 etc. To be honest, as these will be user generated, I can't guarantee that they will be a known length. What if the user uses something like 12439.122. I would prefer if it wouldn't break.

I don't understand enough about bits and bytes and chars to know what the heck I am doing wrong. I understand that I am doing *something* with each individual character that I retrieve. How do I parse this? I have tried using a CSV format, but couldn't figure out how to parse it correctly. That seems to be the most straightforward way of doing this. I would simply read from the SD card until I hit a comma, and then insert that value into the array element. No luck though.

This is why I tried using each number on a separate line like this:

(this is what SENSOR.TXT currently looks like)

Code: [Select]


0.00
0.08
-0.10
0.56
1.36
1.89
3.15
3.15
5.23
6.20
7.34
9.71
9.29
10.76
13.43
17.42
16.76


It seems like my code is even picking up the CR in the text file. Ack, I am a bit lost now. Anyone able to help guide me do something like a CSV parser using my code as a base? I was thrilled to get it this far...

...now I am stumped.

Thanks again for your help.

Ryan

[PaulS]

The data is being put into the array correctly. The problem is that you are printing the index, i, as though it was a printable character. It isn't.

Change the type of i from uint8_t to int, and see if that doesn't look better.

[81Pantah]

This still isn't working correctly.

I am currently running this code. It reads in a text file in CSV format containing values like 0.01, ?0.12, 23.98, etc. I am trying to put each value into an array. Right now, it seems like I am putting each char into the array. Current code:

Code: [Select]


#include <SdFat.h>
#include <SdFatUtil.h>
#include <string.h>

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

// store error strings in flash to save RAM
#define error(s) error_P(PSTR(s))



char responseArray[] = {};
   

void error_P(const char* str) {
  PgmPrint("error: ");
  SerialPrintln_P(str);
  if (card.errorCode()) {
    PgmPrint("SD error: ");
    Serial.print(card.errorCode(), HEX);
    Serial.print(',');
    Serial.println(card.errorData(), HEX);
  }
  while(1);
}

void setup(void) {
  Serial.begin(9600);
//  Serial.println();
//  Serial.println("Type any character to start");
//  while (!Serial.available());
  Serial.println();
 
  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!card.init(SPI_HALF_SPEED)) error("card.init failed");
 
  // initialize a FAT volume
  if (!volume.init(&card)) error("volume.init failed");
 
  // open the root directory
  if (!root.openRoot(&volume)) error("openRoot failed");
 
  // open a file
  if (file.open(&root, "CR.TXT", O_READ)) {
    Serial.println("Opened CR.TXT");
  }

  else{
    error("file.open failed");
  }
  Serial.println();
 
  String receivedData;
 
  // copy file to serial port
  int16_t n;
  uint8_t buf[7];// nothing special about 7, just a lucky number.
  while ((n = file.read(buf, sizeof(buf))) > 0) {
    for (int i = 0; i < n; i++)
    {       
      responseArray[i] = buf[i]; 
      Serial.print("responseArray[");
      Serial.print(i);
      Serial.print("] = ");
      Serial.println(responseArray[i]);
    }
  }


  Serial.println("\nDone");
 

 
  Serial.println(responseArray[0]);
  Serial.println(responseArray[1]);
  Serial.println(responseArray[2]);
  Serial.println(responseArray[3]);
  Serial.println(responseArray[4]);
  Serial.println(responseArray[5]);
  Serial.println(responseArray[6]);
}

void loop(void) {

}

My output looks like this:

Code: [Select]

responseArray[0] = 0
responseArray[1] = .
responseArray[2] = 0
responseArray[3] = 0
responseArray[4] = ,
responseArray[5] = 
responseArray[6] = 0
responseArray[0] = .
responseArray[1] = 0
responseArray[2] = 8
responseArray[3] = ,
responseArray[4] = 
responseArray[5] = -
responseArray[6] = 0


It's going in there, but it is going in one value at a time.

The last Serial.printlns at the end are telling me that it isn't working. It seems like I am still missing something in the code that tells it when to stop reading the file, consider those values a single number, and put THAT into the array. I need a CSV parser of some sort. There needs to be something that tells it to stop at a comma, no?

I have found strtok and sscanf ... but no example of how to use them when reading from an SD card. I have done string parsing before, but never when it is coming in nonstop from the SD card. That is where I am lost.

Many thanks.

Ryan

I

[robtillaart]

responseArray = buf;

That's why every character comes in one array element.

The problem is in the parsing.

You need to two functions to get things right, something like this one, and the other one is left as an exercise

Code: [Select]


float readFloat()
{
  char buf[10];
  readUntil(buf, ',' , 8);  // read from file until 8 chars are read or until a ',' is found
  float rv = atof(buf);
  return rv;
}


[PaulS]

responseArray = buf;

Where did you see this? I see

Code: [Select]

     responseArray[i] = buf[i];  
Which is clearly copying stuff into responseArray correctly.

You are correct that the problem is not in the reading of the data. The data is read from the card correctly. There is no reason to copy it to another array.

You can either use robtillaart's approach or use strtok() to extract a token from buf, and pass that token to atof() for conversion to a float.

The size of buf should be larger, though, so that you can read an entire record, not just 7 bytes at a time.

If you go with robtillaart's approach, don't forget that the last value in a record may not be terminated by a ,.

[robtillaart]

Haha forum bug

I reread my post but the [ i ] is in the quoted text although without the spaces. It is probably interpreted as an italic(?) flag by the forum engine

Fun