SD card read/write with Arduino

Here's a few general notes that I've been meaning to make for a while.

One thing to remember is that uFat simply overwrites data that is pre-allocated on the card. This needs to be a contiguous file else wierdness will ensue. One way to ensure this is to format the card before writing your proxy file to it. Deleting files from the card can leave holes in the FAT which your operating system will use next time a file is created on the card leading to a fragmented file.

Unfortunately resizing a file isn't a simple case of adjusting a value in the directory structure. It entails re-linking the FAT table chains, which means a lot more code in uFat, which is counter to its design criteria. I think it's a reasonable trade-off. Actually, thinking about it, if people were happy with yet another trade off then we might be able to come up with a small extension that can work with one or two files on a card and set their lengths in a one-off manner... But I digress.

The length of the file indicated in the directory is the raw size of the original proxy file, not the number of bytes you've written. Its size remains constant no matter how you write to the space it occupies.

If you write binary data to the file then you'll no doubt have a client application that understands it.

If you write ascii data then things are a little (but only a tiny bit) harder. When the file is written to and later removed to the computer and viewed in a text editor, there may well be a warning that the file is corrupt (at best) to a refusal to load it at all (at worst). This is to do with non-ascii ranged bytes in the file, as explained in a previous post. Warnings can be ignored on the whole. It's best practice to empty the buffer before writing data to it. You could fill it with character 0x20 (ascii for 'space') but that would mean a bloated file when editing. You could fill with 0x0d (carriage returns), which might be better. I'd go with 0 myself, unless it was making editing hard.

Thoughts encouraged.
C

Nachtwind, and those who want it, my PHP for converting to CSV:

<?php
$packet=array(3,2,2); //the number of bytes to read for each value of the 'packet' of data to be converted to a csv line.

$f=fopen('data.bin','r');
while(!feof($f)){ //repeat until end of file
      $csv_line='';
      foreach($packet AS $bytes){ // loop through the packet and generate the CSV line
            if($csv_line) $csv_line.=","; //append a comma if needed
            $csv_line.=getVal($f,$bytes); //append the next value
      }
      echo $csv_line."\n"; //echo the line
}

//reads $bytes bytes from file $f and convert them to a value
//does this by reading each bite as a character, and converting it to its ASCII value
function getVal($f,$bytes){
      $val=0;
      for($i=0;$i<$bytes;$i++){ // loop for however many bytes we require
            $b=ord(fgetc($f)); // read a character and convert it to it's ASCII byte value with ord()
            $val=($val<<8)+$b; // shift the current value left eight bits (8 bits == 1 byte) and then add the byte we just read to the value
      }
      return $val;
}
?>

You must adjust $packet to reflect the number of bytes used for each value. (This is currently configured to read the data generated from my example in SDWriter)

The magic of joining the bytes into larger values happens on this line: $val=($val<<8)+$b
this shifting of bits is how we join two bytes back into an int
this could also be achieved by multiplying by 256: $val=($val*256)+$b;

val byte2 byte1 example values and how they would be stored in multiople bytes
255 255 as we probably know, 255 is the most one byte to contain
256 1 0 to store 256 or more we need another byte to store multiples of 256. (1*256) + 0 =256
261 1 5 (1*256) + 5 =261
65535 255 255 (255*256) + 255 = 65535 //max value of an int (2 bytes)

you could now add a third byte to store multiples of 65536
this can be done for as many bytes as you need to contain your range of numbers, with only a few more bytes you can store over billions!

sirmorris, I find the functionality of uFat is excellent and the trade-offs are not a problem, especially as it seems we are all really only wanting to log data, I can't really see the need for more functionality. If I wanted to do much more I think I would probably want a full FAT filesystem. But only the future can tell :wink:

Also your method of printing to the buffer looks really cool, I haven't got time to study it greatly now, but it looks like it's exactly what JB needs and wouldn't require my suggested writeString function. In fact I guess I could extend my SDWriter in the same way to add the print/println functionality to it :smiley:

Thanks!

It looks like SDWriter would just pop right on top there!

Great stuff. I'm thrilled that uFat is finding some use.

C

I had to change about two lines and now my SDWriter supports .print() and .println() in the same way Serial does, thanks to sirmorris' genius idea :slight_smile:

The new code is available in the same place at: http://code.jhodges.co.uk/SDWriter/SDWriter.zip

There is a new example in there called Print_Data that creates a CSV style file.

The old example is still there as writing byte values is still supported.

If I may I'd like to offer a couple of comments on the writer.

Having the write function which is used by the print() calls go through a general-purpose write function is less efficient than making things work the other way viz:

void SDWriter::write(uint8_t theByte)
{
  sector_buffer[buffer_pos]= theByte;
  ++buffer_pos;
  if(buffer_pos==512)
  {
    flush();
  }
}

void SDWriter::write(unsigned long val,int bytes)
{
  for(int b=bytes-1;b>=0;--b)
  {
    write((val>>(8*b)) & 0xff);
  }
}

This way the print calls all go through the minimal version.

Good point. I guess it is the little things that count when you only have a little processor and memory :wink:

The library is updated with that change.

Thanks again.

You guys are AMAZING!! :smiley:

I've only had time this morning to quickly compile & try bobemoe's Print_Data example, but I can see that it's EXACTLY what I was after. I should be able to incorporate this code to 'format' the data stream in the most user-friendly way possible, which will make life so much easier for my workmates (and me!).

I'll be away from my computer today so I won't be putting much time into this, but in the next couple of days I'll let you know exactly how things have gone.

Charlie & bobemoe - you're work here will be extremely valuable to many others like myself I'm sure; we can't thank you guys enough.

Oh, one last thing - Charlie, did you have any thoughts on why my RTC data is being 'corrupted'?

Thanks again everyone!!

JB

I've looked at the code and there are a couple of things that I'd like to go over with you - as they're application specific maybe you should PM me and we'll take it offline via email. We can post final findings but the interim comms might be a little noisy for this topic.

Is there any hope on getting the read from SD to work?

Hi

Reading from SD is a solved problem. The discussion happening here was to do with a specific problem sketch.

Charlie

But does it read/write FAT? Will it be added to the official Arduino library page?

Hi

Kind-of, and no. The library is intended for tight memory situations where you are willing to accept limitations in usage. uFat allows you to find the starting sector and length of a file on a fat12 or fat16 formatted device.

You can then read data from and write data to those sectors. It doesn't understand the filing system more than it needs to, which in this case is only reading the boot sector, partition table and directory.

In most cases this has been quite enough for me!

Charlie

A few posts ago bobemoe posted code to use the Arduino library print routines to output strings to an mmc card. I've spoken with him and used his code as a basis for DevicePrint. It has been modified to fit my usual criteria - small & tight with few (if any) dependencies.

Get it herehttp://arduinonut.blogspot.com/2009/02/libraries.html along with some of my usual written drivel about whatever and stuff.

Commentary and questioning welcomed as usual. Enjoy.

C

wow,that looks very interestering, gonna have to finish soldering my sd card thingy ;0)

Anyway - is there a chance you could also write a function for reading stuff from an uFat File?

Hi again everyone!

After numerous emails, pulling of hair & re-writes, sirmorris has helped to get my SD card logging dreams up & running. You can get the 'final' sketch here: http://docs.google.com/Doc?id=dqhc6fg_0gmk96kdd. It's certainly not the prettiest sketch, and I can't claim much of it at all as my own work, but it does what I hoped it would (although real-world testing hasn't happened yet).

Here's a photo of my current test setup...


Needs a LOT of work before it's ready for final implementation!! ;D

Please feel free to suggest code improvements/rip my sketch to shreds - I'm no programmer, I've got thick skin, and any help is greatly appreciated!!

One last thanks to sirmorris & bobemoe for their work here - it's amazing how generous you've been with your help & knowledge.

I'll pop a note in here when I get something up in the exhibition area.

JB.

Hi Sirmorris.. unfortunately i dont have much luck compiling your code - i am using 011 of the IDE with a 328 Chip..

Errors are:

In file included from C:\Documents and Settings\Administrator\Desktop\arduino-m328-win_current\arduino-0011-m328\hardware\cores\arduino/WProgram.h:6,
                 from DevicePrint.cpp:1:
c:/documents and settings/administrator/desktop/arduino-m328-win_current/arduino-0011-m328/hardware/tools/avr/lib/gcc/../../avr/include/avr/signal.h:36:2: warning: #warning "This header file is obsolete.  Use <avr/interrupt.h>."
In file included from DevicePrint.cpp:3:
/DevicePrint.h:20: error: expected class-name before '{' token
23: error: microfat2.h: No such file or directory In function 'uint8_t proxyWriter(const uint8_t*, long unsigned int, uint8_t)':
 In function 'void setup()':

As it stands the libraries don't compile under 11 - the Print functionality hasn't been abstracted from the serial library in this version.

You'll need to upgrade to version 0013. http://arduino.cc/en/Main/Software.

It also looks from this error that the mmc, microfat and deviceprint folders haven't been copied to the folder

arduino-00XY\hardware\libraries

which is where they should live.

I hope the upgrade won't cause you too many problems, but there are issues with some sketches - the libraries changed significantly during the 11-12 transition. You can keep both versions side-by-side though, so if you have a problem sketch just compile it under 0011. For all new work, start out in the 0013 IDE.

If you do need to port code then the rule seems to be - if it's an .h file then add

#include <WConstants.h>

and if it's a cpp file, add

#include <WProgram.h>

Good luck. If you need any help just yell.

C

well... i thought it had to do with 0011.. alright.. so i upgraded.. lets see if it works now ;0)

So far it compiles and runs - did i miss something or do you use the same wiring as on the first post shown? Either that has changed or my Card reader is not soldered properly ><

Just for the record - everything is working fine with 0012. Will upgrade to 0013 soon to keep up with the community!

Hey guys, I've been following this very helpful thread but haven't made much progress. I've figured out it's not detecting the card is in since I get STA_NOINIT and STA_NODISK from the MMC init function. I'm using the 012 arduino software.

I've measured the pins with a multimeter and I suspect one of the ground pins isn't connected quite right. So I was wondering how you guys have it hooked up? I bought just the SD card holder and stuck a breakaway thing with the side with the long pins going in to the holder and wires soldered to the short pin side as shown below. I'm using a seeduino which has a 3v3 switch and it's hooked according to the diagram in the OP.

edit>oh right and the question: how do you guys have your card holder hooked up? Is there a better way of doing it? I could have gotten the card holder with a pcb soldered on from sparkfun but it was $15 but I want to use it in a final pcb design too. If nothing works I guess I'll have to suck it up and buy that sd holder board.

Any help would be great!
Thanks!