Micro SD; read from anything else than .txt?

Hi,

Reading a huge amount of data from a .txt file does not seem possible.

I have almost 680,000 numbers in an excel file right now. Is there a way to access the numbers individually using Arduino? I don't think Arduino's storage would allow storing all of the numbers in a matrix, does it?

I am trying to use a micro SD card for storing the data but can't figure this problem out.

Thanks!

You've not said which Arduino you are referring to, as they have different amounts of RAM available. Although I don't think any would be able to hold that much in RAM all at once.

What are you trying to do with the numbers? Generally, you read in a small subset of the complete file to work on.

Hello Mark, thanks for your response.

I will be using an esp32, it has a big RAM but I doubt it can store my data.

Yes, I need to read 1 cell at a time. But I can't address the cell if the data are in a .txt file.
A 170*4000 matrix in excel is what I have.
Is there a way to contact an excel on a micro SD card?
Or should I just try to write a code to address data in a .txt file?
Or anything I'm not aware of?

Thanks again

Storing that much data should not be a problem. Of course it depends on the nature of the numbers.
I wonder how you intend to process it???
Can you give us more general information about your project?

1 Like

Please explain what you are using the numbers for.
An ESP32 should be able to easily read a csv formatted file with 170 numbers per line and 4000 lines. Picking a number from a specific position is not a problem, although it might be a bit slow since you would need to read through the file sequentially (unless every line is exactly the same length and every number takes exactly the same number of ascii characters). Are all the numbers the same format (integer, floating point, etc)? If so, then storing the file as raw data makes random access easy.

Hello Mr. Errington, thanks for your response.

Each of the cells contains a two-digit number.
I read them one by one and compare them to a specific number.

Hello David, Thanks.

You mean I can store the .csv file on the esp32 itself, and use it as a matrix?
My data are not different in size (all two-digit numbers like 12, 50, ...)

Not sure if there would be enough memory for that. If each number is two digits, that could be stored as a single byte, needing 680,000 bytes of memory - I'm not familiar with how much is available on an ESP32. If you stored the data as bytes in a file, then you can use the seek command in the SD library to read a byte from a specific position within the file.

1 Like

Not sure if there would be enough memory for that. If each number is two digits, that could be stored as a single byte, needing 680,000 bytes of memory

I must try to store the data as

byte Matrix[170][4000];

in the code, right?

If you stored the data as bytes in a file, then you can use the seek command in the SD library to read a byte from a specific position within the file.

I will also try to use the seek command in SD.

According to the ESP32 product selector, it looks like the maximum amount of RAM you can get on board is 520Kb. This isn't enough to store the amount of data you have - even as raw bytes.

What are you intending to do with the data in the CSV file? Can you provide more detail than:

With a 2 digit number (which I assume, possibly wrongly, is decimal), that only gives you 100 unique numbers to compare against.

Hi Mark. Thank you for responding.

I see; that won't do then. Thanks.

For an instance, I want to know which of the numbers in my dataset are equal to 15. If any was equal to 15, I would save its row and column.

There may be many 15s in my 170*4000 matrix.

Could I throw light on my problem?

Would appreciate your guidance, thanks!

Ok, next question. How frequently do you need to search the dataset? Is it once a second, once a minute, once an hour etc?

If you don't need to carry out the search very often, say with 10s of seconds intervals, then you could simply read in each row of data from your csv file and do your comparison before moving on to the next row. That way you only need to hold one row in memory at any one time.

1 Like

Well, I need to read a thousand of them in less than a second. But the next thousand can wait🤔
I want to try this approach,

but how can I do this?
Where should I store the csv file and how should I read one row from that?

Throwing this out there,
Is there any possibility to split the data file into say, 10 total files? Then you can use a smaller matrix, scan the data, and move on to the next file...

I don't have too much experience with the SD+arduino, apologies if this is irrelevant

Hi @JBornstein90, thanks for your comment.

Yeah, after Mark's comment, I realize that my dataset can be split into almost 300 files!
Do you have an idea about how I can scan a part of my data and store them in a matrix?

Can you expand on this:

Are you saying that you need to search through the complete dataset 1000 times in 1 second? Or are you saying that you need to read in 1000 rows in 1 second, or 1000 values from the CSV file in 1 second?

Is there any order to the contents of the CSV file(s) that would allow you to discount a part of it?

1 Like

If you structure your file for example to have a matrix of
170x1000
(Whatever size you deem neccessary)
Then you can create the logic needed to

Open File1
Read through array 170x1000
Locate the max values of the set and save them as max1, min1, etc1.

Open file2
Read through array 170x1000
Locate the max values of the set and save them as max2, min2, etc2.

So on so forth... you could also save the max values into their own fixed array if you know the expected size, or you can increase the array size for every file opened... I think you have a few different options that would work here

1 Like

From the 680,000 data I have, I want to scan 1000 of them in less than a second; like datum#1 until datum#1000. But datum#1001 until datum#2000 are not to be read instantly after reading #1-#1000, they can wait and be read afterward.
Therefore, I can store #1-#1000 in an array, and process them. And store #1001-#2000 in that array after that.

Yeah, that's right.
I can make lots of files which are to be read and stored in an array one at a time.

My major problem is that I can't find the way (the code) to read the csv files from my SD card.

With the smaller array size, are you not able to save that as a formatted txt file that is delineated by commas? Does this method still not work?