Working with large organized text files / JSON files? Do you open it entirely?

Projects General Guidance
1

Hello everyone.

I'm looking into reading data from a pretty large text file. Currently I'm looking at JSON because of how infinitely versatile it can be for storing data, I can nest data as many times as I want and traversal is a piece of cake. The question is, will an arduino Due have to store the entire JSON in it's memory when loading it from the sd card? Or can we load only chunks of the json or even search through the json file to get the appropriate data without loading the entire thing into memory?

How big is the JSON file? About the size of an average length King Novel. So yeah, it's gotta be on an sd card.

So the JSON file would have a lot of keys that I'd have to address, call them chapters or sections. There's a lot of them. One thing I could maybe do is have two json files. One is the core large text piece and another one is like a table of contents. An Index of a sorts that would store the lines or some sort of a locator coordinate of where the appropriate key is located. These are just thoughts and I may be re-inventing the wheel here.

2

If you know the lowest key you're interested in, (as in 'b' would be the lowest key in {'a': {'b': 'hi'}}) you can use File.available() and File.read(buf, len) to search for that key. Once the key is found, you can File.read() until the data for that key is completely read into memory.

There are probably better ways of doing it, but that should get you in the right direction.