Go Down

Topic: The faster way to write to an SD card (Read 750 times) previous topic - next topic

adwsystems

I need to write small data sets to an SD card. Is it faster to use file.write() to write each data point and delimiter or should I write the data to a string and file.write() the single string to the card or does it not matter (both are equally as fast)?

Lucario448

I would say it's a hair faster writting the data in a buffered (string) manner. Altough how fast it depends on when the library actually updates the data on the physical media (SD card).

Unlike EEPROMs, SD cards and any other kind of "mass storage device" works by reading/writting data in blocks; usually with a size multiple of 512 bytes. In case of an Arduino, data isn't actually written until one of the following happens:

  • You call either File.close() or File.flush().
  • You surpassed the 512-bytes block boundary (this implies updating the current block and reading the next one, plus some filesystem overhead). This ocurrs automatically, as in code you don't have to worry about it; but still it's important to know because this is the reason why some write attemps might be a bit slower than others (data rate is inconsistent if you don't work in blocks multiple of 512).


In summary: it's a hair faster writting as a string; but if timing is important, you might perceive inconsistent data rates due to block boundaries.

PaulS

Quote
I would say it's a hair faster writting the data in a buffered (string) manner.
It is NOT, and can actually be slower doing it that way.

Data is transferred to a byte buffer, which is written to the file when it gets full. As long as there is room in the buffer for what needs to be written, the data is not actually written to the file. When there is not room, or when the close() or sync() methods are called, the data is actually written to the file, and the buffer (and internal index) are reset.

If you try to pre-format a string, the odds of the string not fitting in the buffer go up.
The art of getting good answers lies in asking good questions.

adwsystems

So the conclusion is? As series of file.print() one for each item, or sprintf to a char[] and a single file.print()?

PaulS

Quote
So the conclusion is? As series of file.print() one for each item
Yes. Less memory, and the odds of there being room in the buffer are better.
The art of getting good answers lies in asking good questions.

Lucario448

If you try to pre-format a string, the odds of the string not fitting in the buffer go up.
Of course, but I was assuming a string no larger than 512 bytes.
Also... don't you think writting data byte by byte or in blocks will have the same effect anyways?

I said it's a "hair" (probably insignificant) faster considering the tiny overhead of calling and returning from a function; however, due to how the Print class works, writting an array will still do multiple calls. In conclusion: I was wrong (unless the function were overriden by a more optimal one); it actually introduces an extra call-return overhead.

But again, performance decrease is insignificant, if not null; so do it the way you feel more convenient.




So the conclusion is? As series of file.print() one for each item, or sprintf to a char[] and a single file.print()?
Yes. Less memory, and the odds of there being room in the buffer are better.
Since I admitted I was wrong from the beginning, I also agree with PaulS. The conversion from "primitive data" to text is unavoidable, and allocating a temporary array is just an extra step.

adwsystems

This is all good information. I was trying to avoid having one sprintf statement in the entire program, then found out there were three other places. Since the data is stored in program variables using integers inclusive of x10 or x100 multipliers to avoid using floats, the sprintf is easier to extract and record the real value.

Thank you for the warning on the 512 byte limit. For a project where I haven't reached the SD logging portion, this will become critical. Is there any way to "monitor" the empty buffer space available?

Lucario448

#7
Sep 05, 2018, 04:59 am Last Edit: Sep 05, 2018, 05:00 am by Lucario448
I was trying to avoid having one sprintf statement in the entire program, then found out there were three other places. Since the data is stored in program variables using integers inclusive of x10 or x100 multipliers to avoid using floats, the sprintf is easier to extract and record the real value.
If you feel comfortable using sprintf for certain tasks, then go for it.
You can skip the array allocation step by pre-allocating it beforehand; what this means is that you should declare it globally and not locally.


Thank you for the warning on the 512 byte limit.
It is not a limit, rather it's a threshold of having to update a block of data into the SD card.
Doesn't necessarily mean it will happen evey time you try to put 512 byte or more at once; but every time the file's "pointer" (returned by position()) reaches a position multiple of 512.

Think like if the SD card space is divided in sections of 512 bytes (formally called "sectors"); the library can access only one section at a time (due to RAM constrains; the library alone takes up approx. 700 bytes). When it needs to write or read beyond the current section, updates it to the physical card if something changed (never happens when the file is opened as read-only), and then reads the content of the next or previous section. This is the slowest part of intertacting with a file (apart from the opening process), mostly because this can generate between 1024 (twice a section) and 4096 (holy cow 8 times a section!) bytes worth of SPI transactions.

Not only SD cards work this way, any type of flash memory does too. Even if you are willing to write a single byte, the whole section has to be updated; which is very inefficient and wears out the memory faster.
That's why the library needs 512 bytes of RAM just to hold one section, sector, page, etc.; it's more efficient to accumulate an entire string and then update, than updating byte by byte.

The good news is that you don't have to worry about this, as long as you don't call flush() too often or repeatedly opening and closing the file just to write a few bytes.



For a project where I haven't reached the SD logging portion, this will become critical.
Are there other time-critical events to make the writing speed so important?


Is there any way to "monitor" the empty buffer space available?
If you are talking about predicting when the block update will trigger, the procedure will be something like this:

Code: [Select]
unsigned int spaceBeforeUpdate (File file) {
  return 512 - (file.position() & 0x01FF);
}

It will never return zero because at zero will already trigger the block update.

adwsystems

#8
Sep 05, 2018, 11:41 am Last Edit: Sep 05, 2018, 07:02 pm by adwsystems
It is NOT, and can actually be slower doing it that way.

Data is transferred to a byte buffer, which is written to the file when it gets full. As long as there is room in the buffer for what needs to be written, the data is not actually written to the file. When there is not room, or when the close() or sync() methods are called, the data is actually written to the file, and the buffer (and internal index) are reset.

If you try to pre-format a string, the odds of the string not fitting in the buffer go up.
I was asking about the space remaining in the buffer mentioned here. Not the cluster size on the SD card.

Lucario448

It's equal to one sector. Didn't you noticed how many times I mentioned the number 512?

Here the buffer is actually more used as a "cache", because it's always full of data (even when part of it doesn't belong to a file).

adwsystems

The buffer and sector size are the same, and file.position() will return the same remaining in the buffer [cache]?

Lucario448

The buffer and sector size are the same
Affirmative...


and file.position() will return the same remaining in the buffer [cache]?
From here you are wrong.

position() returns the current position of the file's "pointer" or "cursor". This is not the pointer of the object itself, but like a "current working index" value.

The SD library always treats files as binary; what this means is that (conceptually) a file is just an array of bytes, if you put new data on already occupied spaces, the old ones get overwritten. It's also like an array that grows when needed: if you put new data from the very end (appending), the file grows accordingly and the old data still remains.

You can't do [] on a File object; but the concept of the array is very similar. position() tells you what's the current index it's working on; and seek() sets the index to work on (it's even zero-relative or "zero-is-first" like an array does!). The position counter automatically increments every time a byte is read (not peeked) or written, so you don't have to constantly setting it every time you try to read or write something.



What is this all about? Well, you can't have the entire file in RAM to use it exactly as a normal array; but you can load it in parts. This is what the "block cache" does.
So, despite the array (I mean file) may be as large as 4 GB (FAT32 limitation); you only can access (or put) data quickly inside a range of 512 bytes.
For "quickly" I mean "without the library having to do its juggling to reach that portion"; and for "juggling" I mean "update and load" (that takes between two and eight block transfers).

Since the file is loaded in sections of 512 bytes, the value returned by position() is determinant. Every time a position or index multiple of 512 (minus one if it's moving backwards) is reached, the library has to load another section.
What this means is that we can find the position relative to it's current section (block or sector) by doing a modulo of position() with 512. In the example code instead I did a bitwise AND with 0x01FF (511) because the result of that modulo operation is the same as just taking in account the 9 least significant bits (I presume doing it this way is faster).
And finally, to find the remaining space before the library has to juggle, it's just matter of substracting the block's size with the result of the previous operation.

That's how I came up with this:
Code: [Select]
512 - (file.position() & 0x01FF)

adwsystems

That's all great information, but I'm trying to inquire about PaulS statement.

Yes. Less memory, and the odds of there being room in the buffer are better.
Is there a way I check so as to manage filling the aforementioned buffer to prevent overruns?

Lucario448

to prevent overruns
The library changes the current sector automatically according to the cursor's position; there is no way an overrun or underrun can occur. You can even read/write 513 bytes or more at once without any problem.

The issue of filling the cache is not the risk of an overflow, but the little slowdown on the recording of a continous data stream.
If the recording pace is low enough, this side effect is not even considerable; otherwise you may perceive an annoying sluggishness in runtime that may cause mistiming or even sampling (data) losses.


In the end, it depends on how frequent you need to save data.

sar_thak

The standard method for composing a SD card picture for a Raspberry Pi (or whatever other reason that requires composing an entire circle picture to the card) from any unix-like framework (eg Linux or Mac OSX) is to utilize the revered dd(1) utility.

dd has been near, essentially unaltered, since the beginning of time. It has an arcane language structure that is totally extraordinary to each other unix summon, and its screen yield is marvelously unhelpful. While it's working, it stays quiet. You have no clue how quick it's going or to what extent it will take, until the point that it in the long run completes and discloses to you what number of squares came in and went out - and even that negligible data is displayed in a somewhat darken organize.

On my Mac I discovered utilizing dd to compose a Raspi boot picture to a SD card to be moderate and untrustworthy. For reasons unknown it declined to keep in touch with the crude circle gadget (/dev/rdiskn), despite the fact that there were no dynamic mounts. Utilizing the cradled gadget record rather (/dev/diskn) it took an entire 30 minutes to compose a 650MB picture… and, after its all said and done, when I pulled the card and place it in my Raspi, by one means or another regardless it booted the old OS that was on there previously.

At that point I found a beautiful utility called pv, which remains for Pipe Viewer. On a Mac it's accessible in Homebrew (mix introduce pv) and Macports (port introduce pv).

Utilizing this, rather than dd if=osimage.img of=/dev/diskn (30 minutes, recall that), I did pv osimage.img >/dev/rdiskn and it took a little more than 2 minutes, and the card worked consummately. (These orders need to keep running as root, obviously).

pv has a huge amount of potential uses, and is a sparkling case of the quality of the UNIX rationality: a little program that completes a certain something and does it well, interconnecting with different projects standard to improve them. I don't know to what extent it's been around, yet I cherish the way that even following 20 long periods of utilizing Linux I can in any case discover something new and flawless that I know will promptly turn into a very much utilized piece of my toolbox and make life less demanding. What's more, I need to chuckle, since I've recently discovered it was composed by Andrew Wood, an old companion from years back. It would appear that it's been around for a long time, is as yet being effectively kept up. Credit.

Go Up