Storing/retrieving data

Projects General Guidance
1

Hi,

I'm working on a project, and things are working out fine. I'm building a weather station for special applications to mainly monitor wind, and changes in atmospheric pressure, temperature and humidity which to some extent could help the user to predict the upcoming weather. The system is helping the user to decide whether it is time for evacuation of area due to high winds.

Currently it is running of a Mega 2560, with the following sensors:

  • Anemomenter
  • BMP180
  • HT22

Additional there is a Oled 128x64 screen, a DS3231 and a SD card unit.

Every minute the systems write 17 variables to the SD card for logging. I also have a array with the size of 300 to evaluate the mean wind, and make a graph to see the development in wind.

Eventually the main goal is to add a GPS unit and a Ethernet unit to be able to give coordinates to each log line as well as send the data to a server. This is a closed system only interesting for the people monitoring.

Now, as i plan how to proceed with this project, i stand at a question on the best way to store the data. as i would like to access them, to make graphs of several of the stored data with at least 24 hours or 1440 logg intervals. older data is not necessary to view locally.

while it might be that i have written my code using bad practice i see that if i where to save all this data on arrays i will fill up the arduinos memory. I currently (by Arduino IDE) i use 34312 bytes of storage and 3877 bytes of dynamic memory. I have briefly read into retrieving data from SD card and found that this is not the best way to go. So, what is the best way of storing approx 20x1440 = 28800 mostly floating variables?

I see that using a separate EEPROM requires addressing all variables i need to retrieve, this might be the best practice but i would love some input on how to do this the best way, so it is possible to expand this little project in future even further.

2

So, what is the best way of storing approx 20x1440 = 28800 mostly floating variables?

Why do they have to be floats?

3

Well, as far as i have read i need a float if i have values with decimals. But i guess i could store them as binaries and convert them to ascii when i need them displayed. But i must be honest and say i do not know what is the best practice here.

And i have to wait 5 minutes to reply! Awesome! :slight_smile:

4

You will have to learn to live with that 5 minute limitation till you have reached 100 posts :wink: Most of us had to :frowning:

Anyway, you have all that data already stored on SD card if I'm not mistaken, so why store it again? You can easily retrieve it to do your calculations.

Which Arduino are you using? If RAM is going to be an issue, post your sketch and somebody might be able to advise if it can be reduced and how to reduce.

5

Yes, i figured it was a way to retrieve the data from the SD card, i now struggle with pulling the data out. Or i'm struggling to find good documentation on how to pull the data. At the moment i'm only pulling ones and zeros. :smiley: or strange symbols, i guess i'm writing data wrong, or reading them wrong.

6

fesan:
i guess i'm writing data wrong, or reading them wrong.

Yes probably.
Or maybe something else.
But you didn't post your code, so that's another five minutes up the spout.

7

so the example for SD library, ReadWrite tells me to use myfile.println to write and Serial.write(myfile.read()) to retrieve the whole file.

so my sketch looks like this after i removed alot of the variables around:

File dataFile = SD.open("winddata.txt", FILE_WRITE);
      if(dataFile.println(winddata)){
      dataFile.close();
      }
File dataFile = SD.open("winddata.txt", FILE_WRITE);
    while (dataFile.available()) {
        Serial.write(dataFile.read());
    }
    dataFile.close();

i know the print is working as i have a error code going if it fails, i also have a counter to show how many times it has written, and i have taken the SD card out to check the data.

"winddata" is an string build up by 17 variables with comma seperator.

direct print from widdata.txt:

DATE,TIME,wind,minwind,maxwind,meanprint,minmeanprint,maxmeanprint,temp,mintemp,maxtemp,hum,minhum,maxhum,hpa,minhpa,maxhpa
11.06.18,21:48,0.06,0.03,0.25,0.01,0.00,0.01,24.40,24.40,24.50,40.80,40.40,41.10,990.00,990.00,990.00
11.06.18,21:49,0.06,0.03,0.25,0.03,0.00,0.03,24.40,24.40,24.50,40.80,40.40,41.10,990.00,990.00,990.00
11.06.18,21:50,0.10,0.03,0.25,0.04,0.00,0.04,24.30,24.30,24.50,40.80,40.40,41.10,991.00,990.00,991.00
11.06.18,21:51,0.06,0.03,0.35,0.05,0.00,0.05,24.30,24.30,24.50,41.00,40.40,41.10,990.00,990.00,991.00
8

You still didn't post your code, you posted a pointless snippet.

("alot" is not a word)

9

i cannot say i agree with you, the 500 lines of code that is mostly irrelevant for the SD read and write is unnecessary to post for having a look at this. and neither does this forum as it does not allow me to post longer than 9000 characters. But i'll try to post some more code to make more sense.

#include <Adafruit_SSD1306.h>;
#include <DHT.h>;
#include <Adafruit_BMP085.h>
#include <DS3231.h>
#include <SPI.h>
#include <SD.h>

String winddata = "";

void setup() {
if (!SD.begin(chipSelect)){
    error = 1;
  }
}

if ((millis() - sdMillis) > sdinterval) {
    winddata += rtc.getDateStr(FORMAT_SHORT);
    winddata += ",";
    winddata += rtc.getTimeStr(FORMAT_SHORT);
    winddata += ",";
    winddata += wind;
    winddata += ",";
    winddata += minwind;
    winddata += ",";
    winddata += maxwind;
    winddata += ",";
    winddata += meanprint;
    winddata += ",";
    winddata += minmeanprint;
    winddata += ",";
    winddata += maxmeanprint;
    winddata += ",";
    winddata += temp;
    winddata += ",";
    winddata += mintemp;
    winddata += ",";
    winddata += maxtemp;
    winddata += ",";
    winddata += hum;
    winddata += ",";
    winddata += minhum;
    winddata += ",";
    winddata += maxhum;
    winddata += ",";
    winddata += hpa;
    winddata += ",";
    winddata += minhpa;
    winddata += ",";
    winddata += maxhpa;

      File dataFile = SD.open("winddata.txt", FILE_WRITE);
      if(dataFile.println(winddata)){
      dataFile.close();
      sdprints ++;
      sdMillis = millis();
      winddata = "";
    } else {
      error = 1;
    }
  }
  File dataFile = SD.open("winddata.txt", FILE_WRITE);
    while (dataFile.available()) {
        Serial.write(dataFile.read());
    }
10

i cannot say i agree with you

Say what you like - you're the one with the five minute choke, not me.
"Alot" is not a word.
"Allot" is, but it doesn't mean "many".

Of course, you could get around the 9000 character post limit by attaching the code.
Or you could put together a minimal sketch that demonstrates your problem.

11

I'm very sorry for not having English as my native language, and i have some spelling mistakes. While you have several thousands of posts, i have only this. I do understand you if you are getting frustrated over newcomers asking the same stupid questions over and over and without understanding of the forum members need for proper code to be of assistance. Did you take offence over me not agreeing with you?

I actually asked for good documentation on the subject and not help on my code. You telling me what is correct, does not help me learn to get this this correct the first time. Reading and understand does.

thank you for the assistance so far, i'll head to google and continue my research.

12

Store the value as a byte; since there is a definitive pattern, disregard the commas and decimal points.

Example of “11.06.18,21:48” :
EEPROM block 0 contains byte (11)
EEPROM block 1 contains byte (6)
EEPROM block 2 contains byte (18)
EEPROM block 3 contains byte (21)
EEPROM block 4 contains byte (48)

So long as you have a definitive pattern, it should be easy to store and retrieve.

13

fesan:
I have briefly read into retrieving data from SD card and found that this is not the best way to go.

Nonsense. You have obviously been reading too briefly.

There is nothing wrong with using floats, but maybe an int is all you need. You apparently already know how to print data to, and dump files from, an SD card, so where is the real problem?

Writing 17 variables a minute adds up to a fair bit and, when reality eventually prevails, you are likely to find that using an SD is a pretty good idea - and an EEPROM is a pretty dumb one. Perhaps this is a discussion best left until after something has gone wrong, and some user wants the data but was unable to download it for a few days, but that last thing you probably need is the choice of storage determining your choice of data.

It appears that ultimately the on-board recording will be as a backup only, in which event an SD probably makes even more sense.

14

May I ask whether everything else than the writing and retrieving is OK?

I have the same problem, a similar project. And I gave up using Arduino in cases like this because of:

  • SDreaders available here read only SD- which I can find only Second Hand, no SDHC, no SDXC readers (in my case, perhaps you can find).
  • SD writing and retrieving proved to be somehow less intuitive
  • using EEPROMs is nice, but it requires far to much work to fit complex tasks.

Nevertheless, from that experience, my guess is that Rasberry Pi is better in cases of high processing, lots of data and complex user interface.

So I turned to only measure and display the values, using an Arduino nano.

The sensors: DHT22 worked better for me in 5V, while the BMP 280 (brother of your sensor) as a 3.3.V device requires a level translator while used with an Arduino nano (in my case).

The DHT22 has a 2 s re-reading time and is very sensitive to wind.

+++

What I would do is to average the read values and manipulate only data of interest. For instance, if you need only high values of wind - for evacuation purposes, maybe would help to dump values less than a certain limit.

The temperature does not usually change in seconds - if we talk about outside temperature, so it make sense to read and store the values only at a 15 minutes interval or so.

The humidity has little relevance in this case, being influenced by local factors rather than by weather, so reading once in 30 minutes may suffice.

The wind speed values depends on the distance to the ground and with the surrounding obstacles. Therefore, it might be enough to store intervals rather than moment values.

By doing so, you will dramatically reduce the data flow and the processing complexity to only the necessary amount. Then use a simple IF declaration to start processing and storing only in the case the read values are above a decided limit.

Note: the time values are examples, you may decide for longer or shorter intervals, according to the project objective and cirumstances.

15

Nonsense. You have obviously been reading too briefly.

Yes, i see that now. While trying to find data on how to store "large" amounts of data, i came over a site stating that reading from a SD card was tedious work and a EPROM was a far better choice.

You apparently already know how to print data to, and dump files from, an SD card, so where is the real problem?

I do not know where the problem is at the moment, as i use the same method to print as the example, and the same method for reading. Mine prints fine, but reading gives absolutely nothing, until i now removed ", FILE_WRITE" from SD.open command. Stupid mistake!

I'll progress with the focus on storing and retrieving from the SD card. Since I now know this is a ok solution.

May I ask whether everything else than the writing and retrieving is OK?

All else is working fine.

And thanks to the pointers for separate the writing interval of the different values. I have this going already for the sensor reading, but i was lost in head trying to write all data, so i did not have the thought of spreading them over several files to track each sensor by it self and combining them with pulling them back to the arduino.

Thanks for the pointers and assistance!

16

For myself I found useful to write down the data of the problem I want to solve. Then I spend few days or weeks to figure out what is the problem to solve. No joking. Sometimes (read many times :slight_smile: the problem is different from what I assume in the first place.

Another point from my experience is that the simple fact that I have few arduino (and other) boards on my desk does not mean that I must use one of them to solve the problem.

And eventually, in my experience, many "easy" projects from the internet turned out to be rather complicate, because the author omitting to post relevant info or just omitting to say that they actually did not build the project, but just copy-paste it or imagine it on paper.

One of relevant cases in line with the above is exactly using the SD cards (again, from my experience, maybe other were lucky, I do not know) - which I now avoid. EEPROM are better, in my view, but only for a few data, not for bulky flows and not to be used along with complex processing. It is nice to have full control but it takes insanely much time to achieve it.

17

fesan:
While trying to find data on how to store "large" amounts of data, i came over a site stating that reading from a SD card was tedious work and a EPROM was a far better choice.

Well, that just goes to how much rubbish there is on the internet. I guess reply #13 falls in to that category too. I'm sure EEPROMs have their place. I'm not sure what that place is, but it is a minor one and doesn't usually include data logging. You should find the datalogging and dump file examples in the SD area of the IDE have pretty well all you need. I'm not sure where your problem is but it might just be procedural, whereby you are trying to read a file that is open for writing. It is safer to keep the file normally closed.
Pseudo code
loop{
gather data
myFile = SD.open(filename, FILE_WRITE);
myFile.print(data);
myFile.close();

check for serial input requesting dump
if yes{
File dumpFile = SD.open(filename);

while (dumpFile.available())
{
Serial2.write(dumpFile.read());
}
dumpFile.close();
}
}

18

@Nick_Pyner

The question is what is the objective of the project? Does it need a lot of data to process? A sophisticated procedure and a huge amount of data can not overpass the lower accuracy of the sensors's readings and can be detrimental to overall understanding. Good engineering is simple engineering. :-).

I am totally against using SDs to log data. It gives the illusion that gathering thousand numbers is beneficial and high accuracy.

The problem with the SD is also that they have to be integrated into a larger project. They are many factors that influence the final result in such cases, starting for variation in components and up to misunderstanding examples.

When the number of variables increases, the chances to do something wrong increases. SDs are just a way to encourage number of variables as well.

I am speaking from practical experience. It is like in photography: better think before taking photos than struggling to find the best photo in a bunch of 100 photos.

If I need only a few data to store (which is preferable anyway) I dont think that is wrong to use EEPROMs. Or any other solution that fits low data volumes.

They are exceptions of course, like movies or audio.

Briefly, I wont store huge data volumes on SD only because I can. Can turn difficult to do and, more important, difficult to validate.

19

This is just pompous claptrap, and I don't believe you have any practical experience at all. The objective is clear to anybody who can be bothered to read it- 28000 variables a day. If this is reduced by 90%, it is still a lot of data, and who are you to say it should reduced at all? And the photographic analogy is absurd. Pictures get assessed by people, data gets crunched by computers. That may be done in Arduino before storage, but may be better done elsewhere.

20

Nick is correct; you would be storing a HUGE amount of data. To dispute him is at your own detriment in future projects. Using the amount of data you are logging, I would only use EEPROM to create a generic graph. From what this n00bie has read, EEPROM has a specified life of 100,000 write/erase cycles.