Compare logged data on SD card to current reading - £25 for working answer

When storing data on your sd card, don't use a human readable time (year, month, day, etc.). ZThat's very hard for a computer to work with - better use the unix timestamp, normally you can get that through Time.now(). That's an unsigned long, seconds since epoch. Now a 24-hour period is simply a difference of 246060 = 86400 seconds. Later when you import that file in a spreadsheet application it's trivial to have those dates converted into something human readable again for you, while the seconds representation makes processing/graphing easier.

Now as you log every 10 seconds, that's 8,640 lines per day. So the record of 24 hours ago should be 8,640 lines before the current line in the file.

But now, how to get to a certain line? There's no line search function, just a position function. So you have to make sure every line is of identical length, so you can simply calculate where you have to go. Here the sprintf() function comes in play. This can guarantee you to write a single line of exactly the same length every time, making searching for an entry in the file easy.

Your file writing section becomes something like this:

 // This section is for storing values to SD card
  myFile = SD.open("Nautilus.txt", FILE_WRITE);
  if (myFile) {
    char line[26]; // 25 characters plus null terminator
    char temp[7];
    dtostrf(T1, 6, 2, temp); // Convert float to string.
    char press[8];
    dtostrf(L1, 7, 2, press);
    sprintf(line, "%10d %6s %7s", now.unixtime(), temp, press);
    myFile.println(line); // writes a 25-character string plus a \n so 26 characters to the file - that's one line.
  }

This way every line will be 26 characters long, and it becomes easy to read back the parts with temperature and pressure (use strncpy(dest, src + beginIndex, endIndex - beginIndex);), and convert it back into float using atof(). The line of 24 hours ago starts 8640 * 26 = 224,640 characters ago.

For comparison: floats are never exactly the same, use an interval. So:

if (abs(yesterdayTemp - todayTemp) < 0.5) {
  // Less than 0.5° difference = equal.
}

Hi wvmarle,

thank you for the help its really appreciated. Currently going through your answer and I have a few questions

1: Can you explain this part of the code, i get sprintf() from your explination however the parts inside the brackets i'm not sure how each works. sprintf(line, "%10d %6s %7s", now.unixtime(), temp, press);

2: I'm really sorry however i'm not sure on the second part of your answer

wvmarle:
This way every line will be 26 characters long, and it becomes easy to read back the parts with temperature and pressure (use strncpy(dest, src + beginIndex, endIndex - beginIndex);), and convert it back into float using atof(). The line of 24 hours ago starts 8640 * 26 = 224,640 characters ago.

GadgetCanyon:
1: Can you explain this part of the code, i get sprintf() from your explination however the parts inside the brackets i'm not sure how each works. sprintf(line, "%10d %6s %7s", now.unixtime(), temp, press);
[/code]

sprintf() helps formatting string.
line is the destination
then the format string: 10 characters digits; 6 characters string; 7 characters string.
And the parameters: the timestamp (seconds since epoch), temperature, pressure.

More reference on sprintf here, and details on the format string here in this printf reference. Note that printf() as such is not available on Arduino, and also the f parameter is not implemented on Arduino, hence the workaround.

2: I'm really sorry however i'm not sure on the second part of your answer

I fixed the smiley face already...

After reading a line from your file in a char array, you can use that function to distil the appropriate parts.

char line[26]; // Line of data goes in this char array.

char temp[7]; // Store the temperature as string in here.
strncpy(temp, *line + 11, 6); // Read 6 characters from line starting at the 12th character and copy this into char array temp.
float T = atof(temp);

Hi wvmarle,

Ok I am nearly there, so far i have this for writing and then reading the SD card.

// This section is for storing values to SD card
  myFile = SD.open("Nautilus.txt", FILE_WRITE);
  if (myFile) {
    char line[26]; // 25 characters plus null terminator
    char temp[7];
    dtostrf(T1, 6, 2, temp); // Convert float to string.
    char press[8];
    dtostrf(L1, 7, 2, press);
    sprintf(line, "%10d %6s %7s", now.unixtime(), temp, press);
    myFile.println(line); // writes a 25-character string plus a \n so 26 characters to the file - that's one line.
  }


// This section is for reading from the SD card 
myFile = SD.open("Nautilus.txt", FILE_WRITE);
  if (myFile) {

char line[26]; // Line of data goes in this char array.
char temp[7]; // Store the temperature as string in here.
strncpy(temp, *line + 11, 6); // Read 6 characters from line starting at the 12th character and copy this into char array temp.
float T = atof(temp);

  }

I was reading your first post where you said:

strncpy(dest, src + beginIndex, endIndex - beginIndex):wink:

The line of 24 hours ago starts 8640 * 26 = 224,640 characters ago.

not sure how to implement this exactly?

Thank you again for all your help and patience

Basically, get the current file size, deduct that number, and do a seek() to point the cursor at that position and start reading. This should do the job:

File f; // The file with your data in it.
uint32_t fileSize = f.size(); // The total file size.
f.seek(fileSize - 224640); // Go to the start of yesterday's record.
char[27] oldRecord;
f.read(oldRecord, 26); // Read 26 bytes.
oldRecord[26] = 0; // Null termination (may not be needed).

char temp[7]; // Store the temperature as string in here.
strncpy(temp, *oldRecord + 11, 6); // Read 6 characters from line starting at the 12th character and copy this into char array temp.
float T = atof(temp);

That should do. As long as I counted all string lengths correctly :slight_smile: If not it'll be off. Just do a Serial.print() of the string to make sure it's got the correct record and positions.

One error, i have marked when it is in the code

File f; // The file with your data in it.
uint32_t fileSize = f.size(); // The total file size.
f.seek(fileSize - 224640); // Go to the start of yesterday's record.
char[27] oldRecord; // Error highlights here 
f.read(oldRecord, 26); // Read 26 bytes.
oldRecord[26] = 0; // Null termination (may not be needed).

Error

 Not used: C:\Users\Brigh\OneDrive\Documents\Arduino\libraries\SD-master
exit status 1
expected unqualified-id before '[' token

Oops

char oldRecord[27];

Right i have nearly got it using the following from what you have said

 // This section is for storing values to SD card
  myFile = SD.open("Nautilus.txt", FILE_WRITE);
  if (myFile) { 

    char line[26]; // 25 characters plus null terminator
    char temp[7];
    dtostrf(T1, 6, 2, temp); // Convert float to string.
    char press[8];
    dtostrf(L1, 7, 2, press);
    sprintf(line, "%10d %6s %7s", now.unixtime(), temp, press);
    myFile.println(line); // writes a 25-character string plus a \n so 26 characters to the file - that's one line.

File f; // The file with your data in it.
uint32_t fileSize = f.size(); // The total file size.
f.seek(fileSize - 52); // Go to the start of yesterday's record. 60sec ago 6*26=156
char oldRecord [26];
f.read(oldRecord, 26); // Read 26 bytes.
//oldRecord[26] = 0; // Null termination (may not be needed).

//char temp[7]; // Store the temperature as string in here.
strncpy(temp, *oldRecord + 20, 6); // Read 6 characters from line starting at the 12th character and copy this into char array temp.
T = atof(temp);
    


Serial.print("20 sec ago: ");
    Serial.println(T);
    Serial.println("---");  



    */
    myFile.close(); // close the file   
  }

The SD card has the following on it:

21906 21.00
21916 21.00
21926 21.00
21936 21.00
21946 21.00
21956 21.00
21966 21.00
21976 21.00
21986 21.00
21996 21.00

I have used the Serial.println(T) to show if its reading anything. So far i keep getting 0.00. any ideas?

I thought there should be three values per line: the time, temperature (T1) and pressure (L1) values. The pressure is missing from your output.

After the sprintf() do add Serial.println(line); to see what is actually written to the SD card, make sure that's correct.

Likewise after the strncpy() add Serial.println(temp); again to makes sure it contains the correct values. atof() returns 0 when the string is incorrectly formatted and it can't make sense of it.

One problem that I see: you're trying to read the line from character 20, which is where the pressure value should be (I don't see it in your output, so it'll be a string of spaces or so, meaning atof() will return 0). The temperature should be starting at character 11.

right i have narrowed down the issue

 // This section is for storing values to SD card
  myFile = SD.open("Nautilus.txt", FILE_WRITE);
  if (myFile) { 

    char line[26]; // 25 characters plus null terminator 26
    //char time1[11];
    //dtostrf(now.unixtime(),10,2,time1);
    char temp[7];
    dtostrf(T1, 6, 2, temp); // Convert float to string.
    char press1[8];
    dtostrf((L1*10), 7, 2, press1);
    sprintf(line, "%10d %10s %6s %7s", now.unixtime(), temp, press1);
    myFile.println(line); // writes a 25-character string plus a \n so 26 characters to the file - that's one line.
    Serial.println(line);

    
    Serial.println(now.unixtime());

    myFile.close(); // close the file   
  }

I edited one of your line "sprintf(line, "%10d %10s %6s %7s", now.unixtime(), temp, press1);" to see if it would make any difference (I added %10s). that is when i started getting -32516 M~1 below. Before i added %10s that first part of the code did not print out.

this is what i can see on the serial monitor

SD card is ready to use.
    -32516    M~1     23.96   36.70
---
    -32506    M~1     23.92   34.59
---
    -32496    M~1     23.88   36.00
---
    -32486    M~1     23.83   34.59
---
    -32476    M~1     23.76   33.89
---
    -32466    M~1     23.71   33.89
---
    -32456    M~1     23.67   31.79
---
    -32446    M~1     23.62   34.59
---
    -32436    M~1     23.58   35.30
---
    -32426    M~1     23.55   29.69
---
    -32416    M~1     23.51   34.59
---
    -32406    M~1     23.48   34.59
---
    -32396    M~1     23.45   33.19
---
    -32386    M~1     23.42   31.79
---
    -32376    M~1     23.41   34.59
---
    -32366    M~1     23.39   33.19
---

and the following is on the SD card

    -32516    M~1     23.96   36.70
    -32506    M~1     23.92   34.59
    -32496    M~1     23.88   36.00
    -32486    M~1     23.83   34.59
    -32476    M~1     23.76   33.89
    -32466    M~1     23.71   33.89
    -32456    M~1     23.67   31.79
    -32446    M~1     23.62   34.59
    -32436    M~1     23.58   35.30
    -32426    M~1     23.55   29.69
    -32416    M~1     23.51   34.59
    -32406    M~1     23.48   34.59
    -32396    M~1     23.45   33.19
    -32386    M~1     23.42   31.79
    -32376    M~1     23.41   34.59
    -32366    M~1     23.39   33.19
    -32356    M~1     23.37   34.59
    -32346    M~1     23.35   33.89
    -32336    M~1     23.32   31.09
    -32326    M~1     23.28   33.19
    -32316    M~1     23.24   34.59
    -32306    M~1     23.21   33.19

From what i can tell there seem to be an issue with the time being added.

Yes, there are serious issues. Multiple. I've been fighting more with sprintf, never been able to get it right the first time.

The line is much longer than expected (due to the extra %10s part). The number for time should be positive, not negative. Have to use u instead of d in the formatting string to make it unsigned.

 sprintf(line, "%10u %6s %7s", now.unixtime(), temp, press1);

This ought to work but there's something odd with the now.unixtime() function - as shown by the output on the Serial monitor. Instead of a number, you get --- printed. Can you give me a link to the rtc library you use? I can probably quite quickly see what's going on there. There appears to be a type problem there. I'm expecting to get a timestamp as unsigned long, but it appears that's not the case here.

Changed it to sprintf(line, "%10u %6s %7s", now.unixtime(), temp, press1); as requested, now getting:

SD card is ready to use.
     35430         21.97
---
     35440         21.99
---
     35450         22.00
---
     35460         22.00
---
     35470         22.01
---
     35480         22.02
---
     35490         22.02
---
     35500         22.02

the rtc lis this RTC with Arduino | Adafruit PCF8523 Real Time Clock | Adafruit Learning System

and the library is this GitHub - adafruit/RTClib: A fork of Jeelab's fantastic RTC Arduino library

unixtime presumably returns an unsigned long integer. The %d format in sprintf implies an 'int' which on UNO/NANO/etc is a 16-bit integer.
Use %ld.
P.S. There is a warning message from the compiler about this - for example:

C:\Users\Peter\AppData\Local\Temp\arduino_modified_sketch_136698\sketch_jul18a.ino:11:45: warning: format '%d' expects argument of type 'int', but argument 3 has type 'uint32_t {aka long unsigned int}' [-Wformat=]

   sprintf(tmp,"Long integer warning %d\n",ul);

                                             ^

Pete

ok so changing the line to the below has worked

sprintf(line, "%10ld %6s %7s", now.unixtime(), temp, press1);

I have now tried to see the filesize using

File Nautilus; // The file with your data in it.
uint32_t fileSize = Nautilus.size(); // The total file size.
Nautilus.seek(fileSize - 52);
  
//  Serial.print(Nautilus.size());
  //  Serial.println("---");

I have tried a few different combinations of the above however none of them have producing anything only "0"

SD card is ready to use.
1531947710  21.57   56.33
1531947710

0
---
---

That snippet won't work as you don't actually open any file.

File Nautilus = SD.open("Nautilus.txt"); // The file with your data in it.
uint32_t fileSize = Nautilus.size(); // The total file size.
Nautilus.seek(fileSize - 52);

Serial.print(Nautilus.size());
  //  Serial.println("---");

This will work a lot better.

So, after working with GadgetCanyon on this for a while it turns out to be a deceptively complex project.

I originally said I'd do it because at first glance it seemed like a simple 10-minute project, and on a PC recording continuously, it would be. However after asking a few questions about usage, the complexity became evident.

If this were continuous data capture on a PC, it would be pretty much dead simple. You have a stream of data being logged every 10 seconds, and after 24 hours of logging (8640 records), you can start to index a pointer that always points 24 hours behind in time and you can read the data at that point in the stream and compare it to the current reading. The memory usage on a modern PC would be trivial. In and done in 15 minutes!

But this is an Arduino and we can't keep 24k of data in RAM and the file manipulation libraries are not as fully-implemented as we'd like. The easy way to do this would be to follow the concept above: keep a pointer in the SD card data and index it forward as needed. Even on a PC this is a bit tricky when working with files, as opposed to memory streams. Due to how the FILE* methods work, I ended up having to open the file in write mode, save the data, close the file and then re-open it in read mode. This was because of limitations on how the seek() and related API calls work.

Even so, this doesn't meet all the requirements. One of the problems is that the logger can be turned on and off at some times, so you're not guaranteed that between now and 24 hours ago, you would have 8,640 records. Therefore the only thing that will work properly, without changing the logging time to use Unix epoch time, or getting involved in complex calculations involving leap years (what happens if yesterday was February 29?), is searching through the data. Easiest is searching from the beginning, but the time becomes O(n) as the file grows. So ideally, we want to search backwards in time, since the maximum search distance is bounded regardless of file length.
I should probably have done that, since in retrospect it would have been simpler, but hindsight and all that...

@GadgetCanyon: As I mentioned, I wasn't able to test it as much as I'd like, so let me know if you have any issues integrating the code I sent into your main sketch.

In a spreadsheet application the date/time may be shown in human readable format in the cells, the underlying storage is always in UNIX time format, as that's easy to work with for a computer. It's just much more hidden from the user than when working with an Arduino - and that's part of the fun of working with such microcontrollers.

For your search problem, you can of course just start searching line by line from either end of the file until you get to the one you need, but it's not exactly efficient.

If there is indeed the possibility of a gap in the data, what will work a lot more efficient is to take the data 8,640 records ago (where you expect it). If you have been running for the past 24 hours you'll see the exact time stamp you expect, and you're done searching. If not, check the difference between the time you see now, and the time you expect, calculate how many records you are off, adjust your seek position, and try again. Within a few iterations you'll have the required record.

Then, as it's a continuous thing and you want to find yesterday's record every 10 seconds: keep that search position (store it in a global variable). 10 seconds later, you simply read the next record, it's most likely the one you need. If it's suddenly much later a time than you expect, just keep the position and return an error or whatever you think appropriate. It simply means there's a gap in the data, and you will catch up. You will have to account for the difference to not be exactly a multiple of 10 seconds as well, of course.

I'm about to start a similar logging battle - but with SPIFFS (ESP8266 internal), and with monthly logging, so at least a lot less records. But enough work to be done on them, as it's got to be handling data of up to 246 nodes :slight_smile:

That's an interesting algorithm and it's certainly worth considering. In case like this, though unless there's a need for the performance, I'd probably just go with a binary or even linear search. Honestly if it didn't seem so dead simple at first glance, I would probably have just done that and gotten it over with.

That's the thing with one-off applications: since software dev time has such a high cost, it's usually worthwhile to just go with the simplest, brute-force approach that works.

You have to search 8,640 records at least - assuming you start from the end of the file. From the start of the file it gets worse, fast.

A quick search on SD card read speeds gives me a read speed of about 6 µs per byte, that would suggest an SPI speed of >1 MHz. At this speed those 8,640 lines of 26 characters each take 1.35 seconds to read. That is not including any access overhead. Other posts give me a speed of 250 kB/s - that would come to 0.9 seconds.

If this is done every 10 seconds, that'd mean you're spending 10% of the time searching! That'd be acceptable only if it has to be done once (upon startup), after that better keep the pointer and just advance it line by line, pausing when there's a gap in the data. No need to search over and over again, as you know the next record you need is either the next one in the file, or not there.