Go Down

Topic: Hacking the SDuFAT lib (Read 5247 times) previous topic - next topic

ardudillo

Hello, please bear with a rather long post from someone who doesn't really know C++.

I have a simple project, reading some sensors (3 ADC readings) and logging the results. The readings are not too close to each other, once every about 10 seconds or more is fine.

For the logging part, I have tried to use an SD card and the SDuFAT library.

The problem is, the lib has many limitations. One of the most obvious ones is that you can't create files so you have to have a preexisting file. This is not too big a problem, the card I 'm using is about 1.9GB so I just create a 1.8GB file called "data.txt" in it.

A few minor bugs are easy to fix, for example:

1- the lib tries to append to preexisting file "hola.txt". See SDuFAT.cpp, function "int SDuFAT::append(const char* filename)", line 454:

return print("hola.txt", data);

This is fixable by changing it to:

return print(filename, data);


2-  The lib knows where the file ends when it hits EOF, in this case 0x03. The problem is it gets overenthusiastic when deleting as it tries to write an EOF at the start of EVERY block. Try that with a file of a couple of GB and you could be waiting a century or two.

Fortunately, this is also easy to fix, in  SDuFAT.cpp, function "int SDuFAT::del(const char* filename)", line 173, I have changed line:

for(long m = 0; m <= sectors; m++) {

to:

for(long m = 0; m <= 1; m++) {            // or change 1 to whatever block you want to stop at

which makes it write to the first couple of blocks. The example sketch seems to be running fine so far and function "int SDuFAT::cat(const char* filename)" seems not to have trouble identifying the end of file without needing to fill up the start of a few trillion blocks with EOFs.


This, however, is all the low hanging fruit I could find. The real trouble begins now and its name is "int SDuFAT::print(..." family of functions.

I found out by trying to write lines of about this size:

dd/mm/yyyy,hh:mm:ss,AAAA,BBBB,CCCC

roughly 35 chars including the newline. The first few lines take just over a second to write. However, this time increases and eventually cripples the application (in my case, where logging occurs once every ten seconds, the moment writing takes longer than that I 'm out of the game).

To find out what's happening I had a look inside "SDuFAT.cpp", function "int SDuFAT::print(const char* filename, char* data)", line 353, which is what all the other "SDuFAT.print..." redirect to.

A rather major problem is that SDuFAT does not keep an internal variable as a file pointer in order to know where we are in the file. Thus, every time a "print" is used, the lib has to go over the whole file to see how big it is (statement "long file1Length = usedBytes(filename);", line 360).

This is OK for short files but as the file gets longer and longer (and given that the SD card is a damn slow beast) it becomes increasingly slow until the whole thing becomes unusable.

To make matters worse, function "int SDuFAT::println(const char* filename, char* data)", which is what most logging applications would use if the calling function does not bother adding the newline to the passed string, calls "int SDuFAT::print(const char* filename, char* data)" twice, once with the string and once with the newline. This means we read the whole damn file twice to find where we are supposed to be writing.

I tried a quick and dirty fix, passing a string with a newline appended to "int SDuFAT::print(const char* filename, char* data)" from inside "int SDuFAT::println(const char* filename, char* data)". Here is my version:

Quote

int SDuFAT::println(const char* filename, char* data)
{
  // Hackdition 11/9/2010
  // "data" is presumably terminated by 0 so find the last char, dump a \n in its place and shift it one down
  int datalength;
  for (datalength=0; data[datalength] != 0; datalength++ );   // find the position of the 0 in the passed array
  char dataln[datalength + 2];                                // make a new array one larger than the passed one
  for (int i=0; i<datalength; dataln = data[i++]);       // copy the original array into the new one
  dataln[datalength] = '\n';                                // put a '\n' at the penultimate position of the new array
  dataln[datalength+1] = 0;                                     // put a 0 at the end of the new array
  int aux = print(filename, dataln);                          // send the new array to the SD card
  // End of hackdition 11/9/2010
  return aux;
}




This should work and indeed does, until you try passing a big string to it at which point hideous things happen. The board hangs, or keeps resetting. Initially I though it was happening when the string was longer than 512 bytes but I soon found out that it happens for shorter strings too. I eventually realised that duplicating the passed string "data" by copying it to locally created "dataln" was making the board run out of memory and all hell was breaking loose. However, I didn't want to throw in the "String" lib to concatenate "data" with a newline as it would make the whole thing heavier still. An inelegant solution is to delete the "println" functions from SDuFAT.cpp and simply pass a newline-terminated string from the calling function to begin with. Any other ideas would be welcome but don't worry too much about it, it's a minor problem.


Now, to the real gritty stuff, the guts of "int SDuFAT::print(const char* filename, char* data)", the workhorse of the "print" family. As far as I can understand, it does weird things, some of which I 've already talked about but let's have another go.

First, it figures out where to write in the most hideous way possible, by reading the file (line 360).

It gets the passed string "data" and copies a chunk of it (up to BYTESPERSECTOR) to class variable "buffer"(line 378) .  It then fills up the rest of the buffer with EOF (line 385). It then goes and writes the buffer ten times (WTF!) to the card (line 390) at the  position it has calculated initially.

If the passed string overflows  BYTESPERSECTOR, it goes and copies the rest of it into "buffer" (line 404), dumps an EOF after it (line 411), fills up with blank spaces (line 416, why not EOFs like before?) and again writes it ten times to the card (line 421).



So, the obvious questions of a relative noob like myself are. Why are things written ten times? Robustness against failings of the hardware? If so, is it OK to add something like:

#define NUMBER_OF_WRITES 10      // or whatever >=1 you feel lucky with, punk

to "SDuFAT.h" and allow people to try their luck with their hardware if they are willing to risk it for greater speed?


And the most important question, should I try to add a class variable, say "long position_in_file" which gets incremented by, say, "count" after line 390 and "count2" after line 421? Then, instead of calling "usedBytes()" and reading through the whole file every time "int SDuFAT::print(const char* filename, char* data)" is called, it could simply look it up and save a hell of a lot of time. Am I missing something obvious?

mowcius

Interesting read.

I have never had much luck with the SDuFAT library and have experienced a number of the problems you have come across.
I'm not enough of a C person to be able to sort it out really so I will be watching to see what you come up with :D

Mowcius

ardudillo

Hello again. I have butchered the lib and it now seems to behave OK for the job I need it to do. I 'm writing 35 byte sentences at an average of 124 ms without any obvious errors.

However, I can't figure out how to attach a file (zip of the .cpp .h and example sketch). Copy pasting the whole thing seems like a daft thing to do. What should I do?

mowcius

Well you can either find somewhere to link it to, upload it to the playground or something similar...

I'm interested in this though. It will be useful for the project I am currently doing...
SD card storage is such a cheap way to do it.
You just need a few resistors and an SD card socket...
Or if you want then you can just solder to an SD-microSD adaptor which cost nothing (most people have one lying around) then you can use micro SD cards which are often pretty much the same price  :D

Mowcius

ardudillo

Um, how do I add to the playground? I tried registering through the Login form:

http://www.arduino.cc/playground/Main/LoginForm

and my forum username got automatically filled. I pressed register and got some positively sounding screen (don't remember what it said) and that was that. Doing it again, gets me a "Sorry, that Username is already taken. Please press Back and try again..." which indicates I must have registered.

However, looking at:

http://www.arduino.cc/playground/Main/Participate

I read that "Unfortunately for the moment we are unable to use your same username and password from the Arduino Forum".

So, what happened? Should I try registering again with another username? But then, why did I seemingly succeed the first time round?

davekw7x

#5
Sep 15, 2010, 01:55 am Last Edit: Sep 15, 2010, 05:18 am by davekw7x Reason: 1
Have you tried SdFat at http://code.google.com/p/sdfatlib/

I have started evaluating the version sdfatlib20100810.zip and I haven't had any problems with creating directories, creating and deleting files, recursively listing all files on the SD card, reading and writing, etc., etc. on my (home-brew) SD interface card (SPI access mode) connected to an Arduino-compatible (home-brew) ATMega328p board.

(Sample sketches in the library do all of these things and more.)

Tested satisfactorily with 4GB SDHC and  4 GB uSDHC with adapter and 8 GB SDHC (and a few older, smaller devices also).

Never touched the library itself.

It uses real FAT32 files just as the devices come formatted from the factory. (It says that it will do FAT16 files also, but I have no reason  to test them.)

The benchmark writes and reads a 5 MByte file in chunks of 100 bytes.  Data rates varied from a little over 100 kB/sec to a little over 300 kB/sec.


Bottom line: I haven't actually used the SdFat library for anything but testing, and I am wondering if others have any experience with it (good or bad).




Regards,

Dave

mowcius

#6
Sep 15, 2010, 07:58 am Last Edit: Sep 15, 2010, 09:41 am by mowcius Reason: 1
Quote
Um, how do I add to the playground? I tried registering through the Login form:

http://www.arduino.cc/playground/Main/LoginForm

and my forum username got automatically filled. I pressed register and got some positively sounding screen (don't remember what it said) and that was that. Doing it again, gets me a "Sorry, that Username is already taken. Please press Back and try again..." which indicates I must have registered.

However, looking at:

http://www.arduino.cc/playground/Main/Participate

I read that "Unfortunately for the moment we are unable to use your same username and password from the Arduino Forum".

So, what happened? Should I try registering again with another username? But then, why did I seemingly succeed the first time round?

Oh sorry, I should have remembered about that. Because the arduino site is changing (apparently), they have stopped additions to the playground for the time being.

Mowcius

ardudillo

Thanks davekw7x, I will have a look, it's just that I had started mucking around with SDuFAT (all I had at a place with no net connection so was stuck with it) and wanted to see it through.

mowcius, (and anyone else who is interested), just PM me your email and I 'll send you the files.

ardudillo

#8
Sep 15, 2010, 06:26 pm Last Edit: Sep 15, 2010, 06:27 pm by ardudillo Reason: 1
Hmmm, had a look at the sdfatlib but they are using Adafruit's GPS shield. I am using Libelium's microSD module for Arduino. I was hoping one of the two ways the module slots into the Arduino would work (most likely the SPI) but no such luck, I 'm getting a "card.init failed" when trying to run the example sketches. It looks like Libelium's shield is not happy with the sdfatlib as mentioned here:

http://code.google.com/p/sdfatlib/issues/detail?id=1

Unfortunately, Libelium's microSD module seems to be the only minimalist solution for SD cards, as far as the playground is concerned:

http://www.arduino.cc/playground/Learning/SDMMC

all other pieces of hardware are shields for something else which also happen to have an SD reader.

davekw7x

#9
Sep 15, 2010, 08:28 pm Last Edit: Sep 15, 2010, 08:37 pm by davekw7x Reason: 1
Quote
I am using Libelium's microSD module for Arduino


Hmmm... According to Libelium's description at http://www.cooking-hacks.com/index.php/microsd-2gb-module-for-arduino.html:

Quote
The SD socket is connected to the SPI port.

My card is a "minimalist connection" and only uses Arduino pins 11, 12, and 13 for the AtMega328p hardware SPI port and pin 10 for the SD card SS.  I use a 74HC4050 for level shifting the 5V logic signals down to the 3.3V required for the SD. See Footnote.

What SS signal is used on the microSD module?  The SdFat library assumes that it is Arduino digital pin 10.  From what I am seeing in the SdFat source code, that can be changed by editing a file in the library, but I haven't tried it with anything else.

[glow][edit]Uhhh-Oh.... See Footnote[/edit][/glow]


Regards,

Dave


Footnote:
Oh, now I see from one of your reference links:
Quote
The Libelium MicroSd module is unreliable due to the fact it uses resistor based level shifters.  


[/begin opinion]
What a crummy way to design things!  Why (oh, why) would anyone do such a thing?
[/end opinion]

mowcius

#10
Sep 15, 2010, 08:37 pm Last Edit: Sep 15, 2010, 08:39 pm by mowcius Reason: 1
Well it is easy enough to connect up a socket and a level shifter to use an SD card.

I have not had time to look at either library yet but I will do soonish for my project. I have had other current issues like not being able to get parts I need until november which is a bit late...  >:(

davekw7x

Quote
socket and a few resistors
See the edited part of my previous post.

Regards,

Dave

mowcius

Quote
The Libelium MicroSd module is unreliable due to the fact it uses resistor based level shifters.  


[/begin opinion]
What a crummy way to design things!  Why (oh, why) would anyone do such a thing?
[/end opinion]

Yeah I have one which I purchase some time back but have never really used it. I might just steal the microSD card socket off it...

ardudillo

Well, the Libelium Sd module with the butchered lib will most probably do nicely for my current project. For my future projects however, I think this shield from Adafruit looks very promising:

http://www.adafruit.com/index.php?main_page=product_info&cPath=17_21&products_id=243

davekw7x

#14
Sep 15, 2010, 11:57 pm Last Edit: Sep 15, 2010, 11:58 pm by davekw7x Reason: 1
@mowcius:
Quote
I have one

My impression is that the resistor level shifters may work with old, slow SD cards but not with more recent SDHC (and SD Ultra) devices.  I'm thinking that slowing down the SPI clock might work.  I think that if one were to use software SPI (bit-banging) the timing could be made to work.  Of course, this would be even slower.

Note to designers and small manufacturers: 74HC4050 chips in small (TSSOP) packages are available for something like 0.10 US dollars in quantity 10. (From Jameco, for example.)

Note to experimenters and bread-boarders: In the U.S. you can get the DIP-packaged versions for something like 0.30 US Dollars in quantity 1. (From Jameco, for example.)

[/begin opinion]
There are other "real" logic solutions.  You don't have to use a '4050, but, for goodness sake, please don't pollute the world with those questionable (and downright silly) resistor (or diode-resistor) level shifters for circuits that are going to try to run at speeds that are supported by modern SD cards.
[/end opinion]

Regards,

Dave

Go Up