Data on SD card, post partially to the cloud or full

Other Hardware 3rd Party Boards
1

Well I really do not know what category to pick for this question.

I have a diy weather station program that logs every 5 seconds the data to an SD card. This data consists of temp, humidity, pressure, wind speed, wind direction, rain fall last hour, last 12 and last 24 hours, dewpoint, windchill, UV index, light intensity etc.. This information is stored on a SD card every 5 seconds. At midnight a new filename is created and the new data is added to that file.
The plan is to create a user account on one of the free weather sites to upload the collected data.
I am thinking about uploading every 10 or 12 minutes to the cloud. Here is where I hope one of you can give me advice how to upload , let's say, the last 10 minutes readings from the SD card.
Uploading every 5 seconds is a no go in my humble opinion. All the solutions I have seen are a direct upload to the cloud. My native language is NOT english so I might not have used the right words to search for hints, guidance or solutions. Therfor I am stuck.
I am not looking for a copy paste solution, but just an idea on how to establish the last 10 or 12 minutes read form the SD card and send it to the cloud. Afterwards I need to implement some check if the information is send and I hope this will not interfere with the program running to collect the meterreadings and/ or watchdog.
The filesize for one day is about 1.4Mb resulting in around 81 bytes per 5 seconds makes around 5kB per 10 minutes.
I know I do not have any code to share on how I think of planning this upload because I really don't have a clue how to perform this action:

I need to open an internet connection to the server and open the file... and then find a way to extract the last 10 or 12 minutes of data from the SD card, send it over the internet, check if the data arrived, close the file etc. Some sort of for loop? Check nr of entries since 10 or 12 minutes ago, post every single line to the cloud. If this takes more than 5 seconds don't forget to take new sensor readings and so on...

Or do I need to store all data from the last 10 or 12 minutes in variables and send that over to the cloud? I might run out of memory..

Or just send all data every 5 seconds to the cloud?

Or send the entire file at the end of the day to the cloud? With our upload speed of 0.8Mb per second this is not the best option imho.

Sorry if this is not the right forum or place to ask, please feel free to replace or delete this post. But I want to learn and understand how to face this problem.

2

My weather station sends data 11 times a second to a MQTT Broker running on a RPi.

Heck, right now there are 8 ESP32's sending data to and receiving data from the MQTT Broker running on the RPi.

On the RPi, you can use Node-Red to do this, I have a Python program running. The Python program communicates with the MQTT Broker. The Python program uses the MQTT messages to let the ESP32's communicate with each other.

The Python program, also, interfaces with my websites database. My web site allows me to view the data collected and control the ESP32's operations.

3

@Idahowalker, well that is impressif. I am running the esp32 wroom to perform some weather tasks. Is it correct that a MQTT broker runs on a local pc or is that a 3rd party server on the internet?
As I understand, having the internet open all the time gives the possibility to upload whenever you want. But what if the internet connection is not established or internet is down? Does the RPi store the readings and send/post it later or are those readings lost in space?
I forgot to mention that I am using a date and time stamp in the SD log to keep track of readings.

1 Like
4

You can have a Internet public MQTT Broker or, as I have my own internal private MQTT Broker.

5

Up to a point those readings will be lost in space as the Queue buffers get filled at 100 publications per topic. The exception is commands and controls signals to the ESP32's over MQTT. Only the last command will be sent.

The RPi can be programmed to do the actions I want.

6

Thanks for the info, this helps me on how to continue.

7

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.