Micro SD; read from anything else than .txt?

Yeah, I think I can...
Not manually though. There would be 680 txt files.

Have a look for Arduino CSV parser. There are a few examples out there. Some read the whole file in, which won't work in your case. But you should be able to see the actual piece of the source code that reads in a line of comma separated text and converts it to bytes.

Some parsers are complicated because they deal with different data types (ints, floats etc), but in your case you know that all values are integers in the range 0 to 99.

As you know the format of the specific CSV file, you can make assumptions that make coding easier. For example if you have a number less than 10, is it written in the CSV file as 5 or 05? If it's 05 (so all entries are 2 characters regardless of their values), then you also have fixed length records - i.e. every line in the CSV file is the same length. You could at it's most simplest level, read in 2 chars, convert them to a byte, skip 1 char (the comma), and repeat till the end of the line.

There's also a standard c function called strtok() which can be used to parse a null terminated string of characters if the leading zero isn't present.

1 Like

Does the data in the spreadsheet change, or is it always the same?

What do you do with the data once you locate the matching numbers in the spreadsheet?

Great. I will try it now.

Hi David. Thank you.

The data are always the same.

I save its row/column number.

A little late to the party; is that 170 rows with 4000 columns?

Is it always 2 digits? So 2 would be 02?

Hi @sterretje, thanks!

Yes

Well, I couldn't store 02 in excel. It is just 2 at the moment.

If you store as text instead of letting excel decide what it is, you can 'force' 02 etc.

It probably will speed up the reading from the card as you can simply read N characters instead of reading till a newline.

Another approach might be to let excel do the heavy lifting and create a "list" with all locations in the matrix for each number. If you're prepared to use multiple files, you can name them 1.txt, 2.txt etc; each file can than contain two columns, one for the row number and one for the column number. It can also be in one file but that might be a little more complex.

1 Like

Fastest output would come from preprocessing the data into 100 files, so you do not need to do any comparisons. If you want to search the original file every time, then have the ESP32 read the file and create a file with all the numbers converted to binary, one byte per number, that way you do not have to search for commas, convert ascii to binary, etc, every time.

I’m not seeing the problem here…
ESP SRAM can fit ~170K per data sample easily…
Read the single CSV file in blocks of 1000 lines.
Scan, compare & process… step on to the next 1000 lines.
Rinse & repeat.

Even on a smaller processor, you can read that 1000 lines - 100 at a time if that’s the problem…

The only issue as the block size gets smaller is the added overhead of SD read time.

This really has the feel of an X-Y problem.

The file has a comma between each two byte field. But the OP states a text file meaning there are CR/LF embedded in there somewhere. Are the CR/LF always after the same number of two-byte_comma fields?
If so, you can compute any starting point in the file you want, reading randomly. Does that help?
Paul

OP has also stated that at the moment numbers less than 10 do not have a leading 0.

From the information the OP has given, it doesn't sound like he wants to randomly access a particularly column/row location, but rather scan the entire spreadsheet to locate all occurrences of a particular number. The ESP32 has sufficient memory to read in an entire line at a time, parse each of the 170 numbers in that line, then do the comparison, although that would hardly be efficient.

I think you will make things more complicated by using the SD card instead of the SPIFFS file system on the ESP32.

1 Like

Will that support the 680,000 2-digits numbers - basically 680,000 bytes of binary data unless you intend to compress it.

Default partition size for SPIFFS is 1.5 MB.

Hi @cattledog, Thanks.
I didn't know about the SPIFFS file system so far. I will read more about it.

I found the section of the source code for this library that reads a cell of my CSV file, and I tried to change that section so that it gives me the value of a specific cell, but I was unsuccessful. Arduino wouldn't upload the code when I changed the source code. It is probably because it's the first time I am manipulating a library.

I think I will for now break my dataset into reasonable sections and store them in .txt files like you said. Storing them in a binary or string manner is a really great idea! thanks, guys.

Indeed. It's still not at all clear why the OP needs to read this data.

I too am curious about why you want to be scanning 680000 bytes for the number of matches to a two digt number.

If you are going to use SPIFFS to hold a group of text files or perhaps the binary

byte Matrix[170][4000];

You might want to use the file uploading loading plugin tool at https://github.com/me-no-dev/arduino-esp32fs-plugin#installation

There is a tutorial in its use at
ESP32 Arduino Tutorial SPIFFS:8. File upload IDE plugin

1 Like