I need to read a file from SD card into a string.
normally after it finishes reading it restarts the chip.
how to allow saving a large string to search it every time I need it without having to read the file byte by byte?
normally after it finishes reading it restarts the chip.
No, normally, it does not. That only happens when you run out of memory.
how to allow saving a large string
Define large. Is large 100 characters or 10,000,000?
without having to read the file byte by byte?
You can use the version of read() that takes an array and a size, to read more than one byte at a time.
normally after it finishes reading it restarts the chip.
No, normally, it does not. That only happens when you run out of memory.
I understand that.
how to allow saving a large string
Define large. Is large 100 characters or 10,000,000?
22000 character
without having to read the file byte by byte?
You can use the version of read() that takes an array and a size, to read more than one byte at a time.
according to what I know and this:
read()
Read a byte from the file.
22000 character
What arduino are you using? Most of them don't have anywhere near enough SRAM to hold such a string.
22000 character
You are dreaming. No Arduino can hold 22000 characters in memory - not any kind of memory.
Look at the source and header files in the library:
virtual int read();
int read(void *buf, uint16_t nbyte);
There are two read() method overloads.
In my project: I search the file so I read it into array of 22. well this is not the problem.
I am asking for the idea of reading a large string into arduino what memory should I increase ??
my file size is 2.5 MB and I might split it but 1 KB or 2 aren't enough.
[edit] the string is 23*100000 char long
what memory should I increase ??
Perhaps you should be asking the question "What memory can I increase?". For the 328-based machines, the answer is none.
For the Mega, additional SRAM can be added. The Due and Yun already have more memory.
Why do you think that you need to hold all 22000 characters in memory?
he string is 23*100000 char long
According to my calculator, that comes in at a bit over 22,000 characters. 208,000 over.
consider that you have an excel file (CSV) and it has 100000 record with 2 columns and you want to search one column and fetch the other. the row has length of 25 char.
you have one of two options:
1- have an array of 25 char and you shift the array and add every byte at the end or even read 25 byte in a time. when you search in the 1000 record it takes 1 minute minimum and you can imagine the rest of record would take hours cuz you search for the text after each 1 or 25 bytes read.
2- have the whole 2.5 MB in a string and you can search it much more faster than you can imagine cuz it doesn't read from SD more than once.
for the SRAM memory you can add 2 MB at least with one way.
ref: Want 2 Megabytes Of SRAM For Your Arduino? | Hackaday
put i wanna be sure that this the one i want not EEPROM which also can be added and it is much easier.
How about making your records fixed length and sorting on the first column, if you haven't already. Then you can binary search your file to get the record you need without having to read the whole thing into memory.
What are you trying to do?
1st column is the id of the person.
2nd column is his name.
i only can have one sorted.
i wanna fetch the name of the id inputted.
and in the same time search with the name.
that is what i want.
and wildbill may you explain a little what you mean ?
Binary search lets you find things quickly in a sorted array. Wikipedia explains: Binary search algorithm - Wikipedia
You can do the same thing with your file if it is sorted by the Id column, but you need to know how many records there are and you need a way to be able to seek to the nth record. To find record count, you can iterate through the file at startup or calculate it from the file size. Seeking the nth record is simple too, as long as the record size is fixed. If not, you would need to build an index, still on the disk.
I'm not sure what you mean by "and in the same time search with the name." but guessing, I'd imagine that you want to be able to search by name too and thus I'd suggest, if you are committed to small arduinos that you need to build an index on the name column too.
A Yun or Raspberry Pi or a Beagle bone might suit you better.
Hey,
I can't have index on two columns (id is given according to a group nor chronological order) and the raw has a fixed size.
and the number of records is fixed as well.
I like the idea of "Binary search algorithm" but it will only help on finding names out of id.
another function is to find id of someone with the name and that algorithm won't do any good.
what are you proposing for fast search using name??
Easiest solution would be to have two copies of the file, one ordered by name, one by ID. Then you can binary search either of them. You just need to be sure that the two files remain in sync.
even if I could handle the sync,
in the id sorted file: the id is the line number.
but in the name sorted file: How to know the line of a character without editing the assembly of the program?
(i.e. if i will look for name of James I'll need to know in what line J starts and look after it)
The method for id or name can be the same - binary search. Which of the two files you use depends on which attribute you're searching on.
It is critical though that the records be of fixed length, otherwise, you won't be able to seek to the nth record, which is necessary if you're going to implement the search algorithm. You may need to postprocess your csv file to pad/trim the name field to ensure that the records are all the same length.
all fixed length is already done.
tell me how to search a name in 100000 record in that algorithm .