Advice on reading from a large amount of data

Using Arduino Project Guidance
#1

I'm working on a project where I need to access some information about each day for 5 years in the future.

For example, for March 5, 2020:

03/05/2020, 15, 30, 50, 12, 11, 15, 50, 25, 18, 19

I'm using an ESP32, so have a bit of flash memory to work with. The data is read-only, I never need to write to it from the Arduino. I can format it however I'd like before flashing.

Two questions at the moment:

  • what's the best way to store that data? Doing a bit of research, I see that progmem can handle large data, and it writes to flash when the script is flashed to the ESP32. Is that the recommended way to go?

  • and what's the best way to look up the given day from my Arduino script? Is the best way to iterate through each line, and break when the correct date is found? Or is there some better way?

I'll be using a Real Time Clock so my Arduino will know the correct date, and the date it looks up will always be the current date.

Thanks for any help.

#2

wrybread: I'm working on a project where I need to access some information about each day for 5 years in the future.

What is the project?

It seems a strange thing to be using a micro-processor for?

...R

#3

5 years is ~1825 days so lets say 2000 days. Your example is ~20 bytes ie we have a lookuptable of "only" 40KiB. At 100Mhz+ it would be less than 1ms to find a line with your suggested simple algorithm (dont know the efficiency of clock/instruction of the ESP32).

Of course you want to store it in flash, so it stays there even with power off (ESP32 functionality depends on the board, I'm unsure if it could stay in RAM)

If you need fast access you could create an binary tree/index like structure, then you can find your date data in in microseconds.

I have no personal experience of ESP32 (yet) so the answers are not proven.

#4

I am confused about what you are doing.

Are you going to datalog and store information once per day for the next 5 years ? that is easy enough.

Do you have a table that is filled on the first day, then for the next 5 years use the date to read from that table ?

#5

wrybread: - and what's the best way to look up the given day from my Arduino script? Is the best way to iterate through each line, and break when the correct date is found? Or is there some better way?

That'll do it. You could do some calendar math and figure out exactly where to find it, or some approximation that tells you where to start looking. But your way is simple and I suspect plenty fast enough, especially since you're only doing it once a day I assume.

#6

ESP32, you might want to use SPIFFS, especially if the SPIFS are going to be read more than written to.

Link to the ESP32 SPIFF API https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/storage/spiffs.html

SPIFFS is a file system intended for SPI NOR flash devices on embedded targets. It supports wear levelling, file system consistency checks, and more.

You might want to consider using a ESP32 WROVER as BLE can eat up a lot of the available RAM.

#7