Reading a 700+ line, 25kb ASCII text file. What read rate should i expect?
1kb/s10kb/s
100kb/s
1Mb/s
Running on a mega2560
Running on a mega2560
Is "b" meant to be bits or bytes?
bytes. standard driver, nothing fancy in the code. MISO/MOSI shared with a max7219 8x32 LED display (separate CS's, no MISO line on the display.)
less than 1kBytes /sec, I think
Really? Why exactly? Must be rotational latency with a bad interleave ...
1 Like
Without the code and schematic nothing can be said "exactly". Just read the data is only part of the story - you should be able to process or store them.
You try to read 25Kb file to the Mega with it 8Kb memory. Where the data will be stored?
Thus, the question rests on whether it is necessary to read this data faster than 1Kb/s ?
Its scanning the file looking for a random entry in a file of 700+ lines. Only 1 line (max 80 chars) is in memory at a time. The schematic is standard mega miso/mosi pins 50 thru 53.
It takes about 2.5 seconds to scan the file or about 10kb/s. Since its all solid state I figured the data would be available in milliseconds, not full seconds.
But if the expected performance is only ~1kb/s then its doing good! I just had no idea what speed to expect.
Speed will be ZERO when a new 512 byte block of data is needed for your read request. That will be blocking until the buffer is refilled.
Yeah, but how long is the block? Is there an easy way to speed it up? What is limiting factor? Are sd's just slow (what IS the access time? Gotta be byte serial). Is it miso/mosi? Crappy driver? All of the above?
Speed isn't really an issue in this particular app. Its currently "fast enuf". But as a point of curiosity, how can we get more juice outta this par-tik'ular piece o' fruit? Eh matey? < /pirate>
Found this:
So it looks like it's NOT limited by the micro sd itself. its got to be miso/mosi or the driver is slow.
I just told you how long a block is. Remember the floppy disks on PC's? Same identical process, except no rotational delay.
Not bytes, time! It blocks, ok, for how long and why! 
You can speed that up considerably if you do a couple of things:
1)Make every line in the file the same length, so you can use seek()
2)Sort the lines in the file so you can use a binary search algorithm.
SPI is not the only method of accessing data on an SD card, there is also SDIO which can be considerably faster but needs the hardware to support it.
I believe your chart in post #9 shows the normal read times using four data lines at once, which is how cards communicate with your computer. The SPI process used by Arduinos, etc., only use one line. So it's going to be much slower than the chart shows.
I agree with david_2018 that having some way of saving the lines in some searchable order, and all the same length, would let you do a binary search, which should cut down the time by a big margin.
One other approach would be to make sure that the file is saved to consecutive sectors on the SD card, so you could access the sectors directly without having to use the file system at all. If you use the file system, your library will have to go back to the FAT to see which cluster comes next, which takes time. Remember that to read in any data or value at all, your library has to read in blocks of 512 bytes. It's like potato chips - you can't read just one.
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.