SD card characters to variables of type double

Hello,

I have been looking at retrieving the final value from my SD card and storing it into a variable of type double. I am currently using this code:

/*
  SD card read/write

  This example shows how to read and write data to and from an SD card file
  The circuit:
   SD card attached to SPI bus as follows:
 ** MOSI - pin 11
 ** MISO - pin 12
 ** CLK - pin 13
 ** CS - pin 10 (for MKRZero SD: SDCARD_SS_PIN)

  created   Nov 2010
  by David A. Mellis
  modified 9 Apr 2012
  by Tom Igoe

  This example code is in the public domain.

*/

#include <SPI.h>
#include <SD.h>

File myFile;
long lastRecordStart;
long fileLength;
long recordCount;
char power[15];
//char y;

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.print("Initializing SD card...");

  if (!SD.begin(10)) {
    Serial.println("initialization failed!");
    while (1);
  }
  Serial.println("initialization done.");
}

void loop() {
  // open the file for reading:
  myFile = SD.open("pow.CSV");
  if (myFile) {
      fileLength = myFile.size();
      recordCount = fileLength / 9;
      lastRecordStart = (recordCount - 1) * 9;
      myFile.seek(lastRecordStart); // Position the read cursor at the start of the last record
        
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
    char power = myFile.read();//Serial.write(myFile.read());
    Serial.println(power);  
    }
   

     //close the file:
    myFile.close();
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

and the output of it is this:

Initializing SD card...initialization done.
5
4
7
3
.
9
0




5
4
7
3
.
9
0




5
4
7
3
.
9
0




5
4
7
3
.
9
0




5
4
7
3
.
9
0




5
4
7
3
.
9
0




5
4
7
3
.
9
0




5
4
7
3
.
9
0

How would I be able to take these characters and store them in a variable called last_saved_value of type double???

Put the characters in an array of chars as they arrive When all of them have arrived put a '\0' in the next array position to terminate it and make it a C style string Now you can used the atof() function to convert the string to a float

UKHeliBob: Put the characters in an array of chars as they arrive When all of them have arrived put a '\0' in the next array position to terminate it and make it a C style string Now you can used the atof() function to convert the string to a float

Thank you very much! I've implemented this change to the code shown below and it's printing as expected:

void loop() {
  // open the file for reading:
  myFile = SD.open("pow.CSV");
  if (myFile) {
      fileLength = myFile.size();
      recordCount = fileLength / 9;
      lastRecordStart = (recordCount - 1) * 9;
      myFile.seek(lastRecordStart); // Position the read cursor at the start of the last record

      String powerstring = "";
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
    char power = myFile.read();
    powerstring = powerstring + power;
    }
   lastsavedvalue = powerstring.toFloat();
     //close the file:
    myFile.close();
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

Not quite what I suggested as I had in mind to use a C style string, ie a zero terminated array of chars, but as long as you are happy

UKHeliBob: Not quite what I suggested as I had in mind to use a C style string, ie a zero terminated array of chars, but as long as you are happy

Yes it seems that I have jumped the boat in terms of my excitement. I may have misunderstood what you meant, may you provide an example if that's possible?? Also I am now encountering an error in the way I am reading certain values stored in my SD card:

Current Charge:5464.56
Last saved value is: 3.00
Current Charge:5463.65
Last saved value is: 63.65
Current Charge:5462.84
Last saved value is:5462.84
Current Charge:5461.95
Last saved value is:5461.95
Current Charge:5461.03
Last saved value is: 95.00
Current Charge:5460.08
Last saved value is: 460.08

It seems that the amount of bytes of a single reading varies, thus making my calculations noted here invalid:

 if (myFile) {
      fileLength = myFile.size();
      recordCount = fileLength / 9;
      lastRecordStart = (recordCount - 1) * 9;
      myFile.seek(lastRecordStart); // Position the read cursor at the start of the last record
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
    char power = myFile.read();//Serial.write(myFile.read());
    Serial.println(power); 
    }
     //close the file:
    myFile.close();
  }

It sometimes varies between 7 and 9 bytes and I am not sure why as the number of digits remain the same. Could it be the way I am converting my string to a double? Thanks for the help!

Take a look at Serial input basics - updated

Although the thread is about serial input the principles are the same for input from a file. Robin has examples of how to read a stream of data character by character, put them in an array of chars and when complete, convert them. See example 4 for instance. It will work just as well to receive a float and you can use the atof() function instead of atoi() to convert the string to a float when it is complete

UKHeliBob:
Take a look at Serial input basics - updated

Although the thread is about serial input the principles are the same for input from a file. Robin has examples of how to read a stream of data character by character, put them in an array of chars and when complete, convert them. See example 4 for instance. It will work just as well to receive a float and you can use the atof() function instead of atoi() to convert the string to a float when it is complete

The forum you linked me was very useful, I appreciate it!
Here is my new code:

// Example 4 - Receive a number as text and convert it to an int
#include <SPI.h>
#include <SD.h>

File myFile;
long lastRecordStart;
long fileLength;
long recordCount;
double lastsavedvalue = 0;

const byte numChars = 32;
char receivedChars[numChars];   // an array to store the received data

boolean newData = false;

double dataNumber = 0;             // new for this version

void setup() {
    Serial.begin(9600);
    Serial.println("<Arduino is ready>");
      while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.print("Initializing SD card...");

  if (!SD.begin(10)) {
    Serial.println("initialization failed!");
    while (1);
  }
  Serial.println("initialization done.");
}

void loop() {
    recvWithEndMarker();
    showNewNumber();
}

void recvWithEndMarker() {
    myFile = SD.open("pow.CSV");
    static byte ndx = 0;
    char endMarker = '\n';
    char rc;
   
    if (myFile) {
    fileLength = myFile.size();
    recordCount = fileLength / 7;
    lastRecordStart = (recordCount - 1) * 7;
    myFile.seek(lastRecordStart); // Position the read cursor at the start of the last record
    // read from the file until there's nothing else in it:
    while(myFile.available())
    {
    rc = myFile.read();
    
        if (rc != endMarker) {
            receivedChars[ndx] = rc;
            ndx++;
            if (ndx >= numChars) {
                ndx = numChars - 1;
            }
        }
        else {
            receivedChars[ndx] = '\0'; // terminate the string
            ndx = 0;
            newData = true;
        }
    }
        myFile.close();
    }
}

void showNewNumber() {
    if (newData == true) {
        dataNumber = 0;             // new for this version
        dataNumber = atof(receivedChars);   // new for this version
        Serial.print("This just in ... ");
        Serial.println(receivedChars);
        Serial.print("Data as Number ... ");    // new for this version
        Serial.println(dataNumber);     // new for this version
        newData = false;
    }
}

And here is my new data because of it:

<Arduino is ready>
Initializing SD card...initialization done.
This just in ... 5114.68

Data as Number ... 5114.68
This just in ... 5114.68

Data as Number ... 5114.68
This just in ... 5114.68

Data as Number ... 5114.68
This just in ... 5114.68

Data as Number ... 5114.68
This just in ... 5114.68

Data as Number ... 5114.68
This just in ... 5114.68

Data as Number ... 5114.68
This just in ... 5114.68

Data as Number ... 5114.68

My concern is with the varying number of bytes per reading. Would this method put my concerns to rest?? I presume the number of bytes per reading will vary as the number of digits would vary:

iqasi096:
Yes it seems that I have jumped the boat in terms of my excitement. I may have misunderstood what you meant, may you provide an example if that’s possible??
Also I am now encountering an error in the way I am reading certain values stored in my SD card:

Current Charge:5464.56

Last saved value is: 3.00
Current Charge:5463.65
Last saved value is: 63.65
Current Charge:5462.84
Last saved value is:5462.84
Current Charge:5461.95
Last saved value is:5461.95
Current Charge:5461.03
Last saved value is: 95.00
Current Charge:5460.08
Last saved value is: 460.08




It seems that the amount of bytes of a single reading varies, thus making my calculations noted here invalid:



if (myFile) {
     fileLength = myFile.size();
     recordCount = fileLength / 9;
     lastRecordStart = (recordCount - 1) * 9;
     myFile.seek(lastRecordStart); // Position the read cursor at the start of the last record
   // read from the file until there’s nothing else in it:
   while (myFile.available()) {
   char power = myFile.read();//Serial.write(myFile.read());
   Serial.println(power);
   }
    //close the file:
   myFile.close();
 }




It sometimes varies between 7 and 9 bytes and I am not sure why as the number of digits remain the same. Could it be the way I am converting my string to a double?
Thanks for the help!

Some context regarding the varying number of bytes saved into the SD card per reading would be greatly appreciated.
Thanks

My concern is with the varying number of bytes per reading.

As long as you don't read more than the value of numChars then you will be OK

What is the source of the data in the file ?

UKHeliBob:
As long as you don’t read more than the value of numChars then you will be OK

What is the source of the data in the file ?

The data comes from a current sensor connected to a lipo battery where the data has been altered to act as a coulomb counter running from 0 to 11000 mAh. The SD card data is there to retrieve the last saved value and adds it to a global variable to show the amount of overall current generated by the battery. So if the battery generates 5400mah and is drained till its empty, it has to add whatever extra mah generated at the charging restart to the 5400 mah. There’s a potential that the final value will be very large, further consideration for this will be taken in the review phase.

Hello,

I have been using myFile.seek() to position my cursor at the beginning of my last value saved in an SD card to store the data into an everchanging variable. This variable would be used in a mathematical calculation whose result would be summed up with the aforementioned variable and stored as a new value in the SD card.

I was able to make a code to read the most recent single additions of 7 bytes from the SD card data here:

myFile = SD.open("pow.CSV");
  if (myFile) {
      fileLength = myFile.size();
      recordCount = fileLength / 7;
      lastRecordStart = (recordCount - 1) * 7;
      myFile.seek(lastRecordStart); // Position the read cursor at the start of the last record

      String powerstring = "";
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
      char power = myFile.read();
      powerstring = powerstring + power;
    }
    lastsavedvalue = powerstring.toFloat();
     //close the file:
    myFile.close();
  }

The issue occurs when reading larger/smaller single additions of data:

-112.04 // 7 bytes
-145.79 // 7 bytes
-35.11 // 6 bytes
39.39 // 5 bytes
-28.30 // 6 bytes

Is there a way to ensure the cursor starts the cursor at the correct final value every time whether the data entry is 1 or 10 bytes??

Is the data stored on the SD card with a new line end marker?

If so, then the varying number of bytes will not matter as you can read to the end marker.

Your code appears to use a fixed byte counter/record length to determine where to start with a file.seek() and and an end marker for where to end.

cattledog: Is the data stored on the SD card with a new line end marker?

If so, then the varying number of bytes will not matter as you can read to the end marker.

Your code appears to use a fixed byte counter/record length to determine where to start with a file.seek() and and an end marker for where to end.

Hello, My issue is determining where to start with file.seek(). My code works just fine for a constant number of bytes, when the data starts to vary is where the problem lies. When the total size of my file changes, it takes the whole file size and divides it by the number of bytes per reading. If the number of bytes per reading varies, my starting position cursor will vary with it. Is there a way to account for such discrepancy to move away from using a constant number of bytes as I have done previously?? Thank you

Ensure that every reading is the same size by padding them. You could use dtostrf for that purpose.

You know how many records have been saved at the time that you need to load the final one, so, read and discard unwanted records then read the final one

Better still, every time you save a record to SD save it in a variable. The contents of the variable will always be the final record written

wildbill:
Ensure that every reading is the same size by padding them. You could use dtostrf for that purpose.

So if I set my values to a set number of bytes (lets say 12), there is a way I can make sure every reading is 12 bytes regardless of the useful number of bytes holding my data?? How do I use dtostrf to do that??

UKHeliBob:
You know how many records have been saved at the time that you need to load the final one, so, read and discard unwanted records then read the final one

Better still, every time you save a record to SD save it in a variable. The contents of the variable will always be the final record written

How would I be able to know how many records have been saved when each record varies in data over time?? are you suggesting that I read a set number of bytes and have them deleted and rewritten once an extra byte is introduced? How would I be able to do that??
I already have the SD data saved in a variable, the SD card is there to carry on the variable’s count from where it left off after my battery powered system runs out of charge so that my variable does not reset and continue from the very beginning.

If it is a CSV file can you not just read to he next ‘,’ or to EOL/EOF?

You don’t show the code for writing the file, but if you are writing <7 chars for any record then the way you are seeking in the file will not be reliable.

How would I be able to know how many records have been saved when each record varies in data over time??

Every time you write a record to SD add 1 to a value held in EEPROM or on a file on the SD.

Topics on the same subject merged

Why did you start a second one ?

Cross-posting is against the rules of the forum. The reason is that duplicate posts can waste the time of the people trying to help. Someone might spend 15 minutes (or more) writing a detailed answer on this topic, without knowing that someone else already did the same in the other topic.

Repeated cross-posting will result in a timeout from the forum.

In the future, please take some time to pick the forum board that best suits the topic of your question and then only post once to that forum board. This is basic forum etiquette, as explained in the sticky "How to use this forum - please read." post you will find at the top of every forum board. It contains a lot of other useful information. Please read it.

Thanks in advance for your cooperation.

So if I set my values to a set number of bytes (lets say 12), there is a way I can make sure every reading is 12 bytes regardless of the useful number of bytes holding my data?? How do I use dtostrf to do that??

Using dtostrf() to create a constant length c-string is somewhat tricky as you need to account for the null terminator. The minimum width specification of dtostrf() does not include the null.

dtostrf(floatdVar, minStringWidthIncDecimalPoint, numVarsAfterDecimal, charBuf);

Here’s an example

float floatVarA = 123.45;
float floatVarB = 123456.78;
char cstrVarA[12]; //size buffer to hold 11 characters incl '.' + null terminator
char cstrVarB[12];

void setup() {

  Serial.begin(115200);
  
  Serial.print(dtostrf(floatVarA, 11, 2, cstrVarA));
  Serial.print('\t');
  Serial.println(strlen(cstrVarA));

  for (byte i = 0; i < 12; i++)
  {
    Serial.print(i);
    Serial.print('\t');
    if (cstrVarA[i] == '\0') Serial.println("NULL");
    else Serial.println((byte)cstrVarA[i]);
  }

  Serial.print(dtostrf(floatVarB, 11, 2, cstrVarB));
  Serial.print('\t');
  Serial.println(strlen(cstrVarB));

  for (byte i = 0; i < 12; i++)
  {
    Serial.print(i);
    Serial.print('\t');
    if (cstrVarB[i] == '\0') Serial.println("NULL");
    else Serial.println((byte)cstrVarB[i]);
  }
}
void loop() {}

countrypaul:
If it is a CSV file can you not just read to he next ‘,’ or to EOL/EOF?

You don’t show the code for writing the file, but if you are writing <7 chars for any record then the way you are seeking in the file will not be reliable.

my apologies… Here is the entire code:

void loop() {
  unsigned long currentMillis = millis();
  if (currentMillis - previousMillis >= interval) {
    // save the last time you blinked the LED
    previousMillis = currentMillis;    
    //Calculate Amperage Average
    average_amperage();
    // Calculate Total Amperage
    total_amperage();
  }
    if (currentMillis - previousMs >= 15000) {
     previousMs = currentMillis;
      //Retrieve last saved value from SD card if arduino turns off
     SD_set();
    Serial.println("Last saved value is:");
   
     averageAmps = totalAmps / (45);
     total_power_produced();
     Serial.print(lastsavedvalue);
     write_data(totalPowerProduced);  //write data
     Serial.println("data written");
     write_counter();
     totalAmps = 0;
   }
}

void average_amperage()
{  //
  points = analogRead(inputPin);
  voltage = ((points / 1023.0) * readVcc()) ;
  averageamperage = (( voltage / readVcc()) * 36.7) - 18.3 ;  

  if (averageamperage > 0 && averageamperage < 0.11)
  {
    averageamperage = 0;
  }
  else if (averageamperage < 0 && averageamperage > -0.11)
  {
    averageamperage = 0;
  }
  // send it to the computer as ASCII digits
  Serial.print("Average:");
  Serial.println(averageamperage);
}

void total_amperage()
{
  totalAmps = totalAmps + averageamperage;
}

void SD_set(){
  // open the file for reading:
  myFile = SD.open("pow.CSV");
  if (myFile) {
      fileLength = myFile.size();
      recordCount = fileLength / 8;
      lastRecordStart = (recordCount - 1) * 8;
      myFile.seek(lastRecordStart); // Position the read cursor at the start of the last record

      String powerstring = "";
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
      char power = myFile.read();
      powerstring = powerstring + power;
    }
    lastsavedvalue = powerstring.toFloat();
     //close the file:
    myFile.close();
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void total_power_produced()
{     
      powerProduced = averageAmps * 3.7;
      myFile = SD.open("pow.csv");
      if (myFile.available()) {
            totalPowerProduced = lastsavedvalue;
            totalPowerProduced = totalPowerProduced + powerProduced;
        } //else {
          // create totalPowerProduced
         //  totalPowerProduced = totalPowerProduced + powerProduced;
        //}
      //close the file:
    myFile.close();
}
void write_data(double power_input)
{

  myFile = SD.open("pow.csv", FILE_WRITE);

  // if the file opened okay, write to it:
  if (myFile) {
    myFile.println(power_input);
    myFile.close();
    flag = true;
    //Serial.println("Write file successful!"); //print out COM Port
  } else {
    flag = false;
  }
}
void write_counter()
{

  myFile = SD.open("counter.csv", FILE_WRITE);

  // if the file opened okay, write to it:
  if (myFile) {
    if(flag == true){
    SDCounter++;
    myFile.println(SDCounter);
    }
    myFile.close();
    //Serial.println("Write file successful!"); //print out COM Port
  } else {
    //Serial.println("error opening test.txt");
  }
}