What is the best way to store lots of sensor data and being able to add new data

Using Arduino Project Guidance
#1

So I’ve got an Ultrasonic sensor that reads a water level and i’d like to have that data displayed on a graph on a webserver (it’s on http://185.96.166.200 (/, /time, /chart)).
Currently I’ve managed to make the webserver take the sensor data and use the Google Charts API to display the data (currently uses placeholder data).

The next thing that i have to figure out is storing that data for the chart to display. My board is a NodeMCU 1.0(ESP-12E) 4M and currently I’ve looked at the following solutions:

  • Using a file on the SPIFFS file system on the NodeMCU board
  • Using an array
  • Using a LIFO stack to store the data

My problem is i can’t fully understand any of these solutions and I searched a lot, but I can’t really understand how to implement it.

So I could use a file and store the data in a text file, but the thing that I don’t understand is how I can make it delete the oldest data after it reaches (228 lines(every 5 minutes in 24 hours)) while still adding the newest number to the bottom. And the other thing is how I’ll be able to add all that data to the html that I store as a string in my program.

The next thing that I could use is an array, but the only thing i’d like to know is is there any way to move the start of an array, so that the program will start reading from the 2nd value there(and have that 2nd value indexed as 0, because I don’t want the array to be huge). At least with the array the adding to the html string will be easier.

The last solution I read about would be the stack, but I don’t have any experience with a stack, so I don’t understand much of it. LIFO would be really good, but the thing is the libraries I found StackArray and StackList, at least from what I understand there isn’t a way to read all the data without having to use the .pop method, but that’d erase that data and I don’t want that. Is there anywhere where I could read more about stacks and how to implement them for something like this?

So which of these 3 solutions would be the best suited for my case and also can you please explain how I could do what I mentioned I don’t know how to do using each of the 3 solutions?

I’ve attached my whole program. There I’ve currently been experimenting with the StackArray library, but I’m not really sure what to do

HelloServer.ino (6.33 KB)

#2

SD card.

#3

how I can make it delete the oldest data after it reaches (228 lines(every 5 minutes in 24 hours)) while still adding the newest number to the bottom.

The existing Arduino SD card libraries do not support such operations.

Memory is very limited on a standard Arduino, but if the water level data can be stored as a byte or an integer, 228 measurements will easily fit in available memory. Use a ring buffer -- there are lots of code examples on line, like this one or this one.

#4

This guide takes small steps towards a temp server that stores months of 5-minute data with NTP timestamp onto the NodeMCU's onboard SPIFFS memory, and display it in a graph.
Shouldn't be that hard to convert to a water level sensor.
Leo..

#5

Use a lot of small files, each with a timestamp as filename (do check on how this works on SPIFFS/SD as you may be wasting quite some space - files tend to take at least one sector).
Then you can read them all, send the data as single data set, and remove the oldest ones when you don't need them any more.

#6

If you have to transmit all the data to the web server, then do that immediately for every single value.

#7

This sounds like an X/Y Problem, you dictate the proceedures and do not seek the end result.

you can post your data, [ post #5] time, value1, value2,… value8, value9
to a cloud service.

then use that cloud service to graph your results.

the graph will soon be too much information to read, so you will only want to graph the last 100 readings or so.

and, the cloud service can do that for you as well.

for you to use google charts, I believe it is easier if you post that running data onto a google service, on the cloud.

There are Arduino graphing programs that will post to an LCD as was noted in post #3
if you want to see more about graphing on a local device, youtube has quite a few examples.

youtube link - http://www.youtube.com/watch?v=5JKFGhiO29E

#8

Thanks a lot for all the replies. I ended up using the CircularBuffer library to store all the data, and every thing is working now. Huge thanks for all your help from everyone.

#9