SD card read/write with Arduino

Downloaded & tried, thanks bobemoe!

Here is what is in my data.bin file after running the sketch:

C:\arduino-0012\output.jpg
-< edit >- bugger - how do I put a photo in here?!? There should be a bmp of strange shapes & symbols instead of numbers here...

Do you know why I'm getting this wierd stuff instead of numbers?

Thanks for your help anyway!!

JB

Hi JB,

You wont be able to open it in a text editor... its saved as binary - the actual bytes rather than an ASCII representation of those numbers.

Each byte (value 0-255) in a TEXT file represents one character. So a file containing the string "123", consists of 3 bytes of value: 49,50 and 51, which represent the characters 1,2 and 3 (see http://www.asciitable.com/)

If we write the single byte value 123 to a file we get a file which consists of 1 byte of data, value 123. But if we view this as TEXT we will see only 1 character, a { in this case, which is represented by the value 123. Which is why you are seeing funny characters, as a lot of the ASCII set is symbols and other weird stuff!

I will add a writeString() function to the library, but I think you would probably have trouble converting your byte's and int's to a string, I did look into this briefly (something to do with stdio.h, sprintf() and bigger code!) but quickly decided I will stick to writing values not strings to the file.

I wrote a simple PHP script that converts the raw data into a CSV. What operating system are you using? I can share this with you, and help you with it if your on Mac or Linux, but wouldn't be able to help so much if on Windows.

I understand you want relatively user-friendly access to your data. This conversion process could be packaged into a single command or double clickable file, data.bin goes in, data.csv comes out.

It seems you are at a bit of a fork in the road, and I am not sure the best route for you. What does everyone else think? I thought string manipulation looked a little tricky in Arduino/C. I've only had my Arduino a few weeks, also I am more used to PHP/Java/BASH. Is this conversion a sensible trade-off for avoiding strings in this case?

If you were writing lots of data, and fast, you would most certainly want to write byte values rather than ASCII strings, as this will keep file sizes much smaller and reduce the amount of time spent writing to the card (exactly what I need).

In your case trying to figure out this string thing (its actually an array of chars) might be an idea.

Let me know which route you choose.

I will add the writeString() function. Can anyone link us to any good tutorials on strings and sprintf?

Thanks for explaining the weird characters, your post is very informative!!

Unfortunately (for me) I'm using Windows... don't get much choice with the Government!! ;D

I guess that's another option to look at; does anyone know of an existing Windows program which could convert the raw data to CSV or just plain text? I had hoped to have been able to just pull the SD card out, plug it into the laptop & view the output in 'real' numbers, but I now understand that I can't do that currently. I can't get over how generous the Arduino community is in terms of writing & sharing code with people like me, and I would be deeply grateful to have a look at a writeString() function if you were to play around with one!

Since my data is only taken once a minute, and the capacity of SD cards provides essentially unlimited storage in my case, speed and file size are not really an issue. Your comment:

In your case trying to figure out this string thing (its actually an array of chars) might be an idea.

...has got me thinking though, so I'll start playing around with that & see where it takes me...

Thanks again!

I could help you by writing a small Windows script or program for it.. sooner or later i will have to do that anyway.

I think i got the basic idea of how those values are written but bobemoe could you though give me your php code so i could simple rewrite it for windows?

I think the way to go is to use the code already present in the libraries. Check out .\hardware\cores\arduino\Print.cpp

This base class provides the 'printing' functionality. Since you're already wearing the cost of the code whenever you use the Serial library it can be used virtually free. All you need to do is provide a subclass which provides a write byte function which puts a byte to the MMC buffer and writes the sector when necessary.

What you get is the ability to write strings using the exact same functions as you'd use for Serial! How cool is that.

As a super bonus, you could interchangeably use the MMC output with the Serial output for testing..

Here's how. (Please forgive typos, I'm away from arduino right now so I can't test)

class MMCWriter : public Print
{
  public:
    void begin( .. )
    {
       // initialise the mmc byte-by-byte writer here
    }

    void flush(void)
    {
       // flush the buffer to the card here
    }

    virtual void write(uint8_t theByte)
    {
       // call the function that puts a byte in the buffer here 
    }
};

MMCWriter MMCio;

void setup(void)
{
  MMCio.begin();
  MMCio.println("Hello! This string will go to the buffer!");
  MMCio.println(123, DEC);
  MMCio.flush();
}

That's it. I'll leave the serial/mmc swaperoo trick for another post (or you could work it out if you've got 5 mins).

There's a huge amount of power in the arduino core - take some time to have an explore and who knows what you might find :slight_smile:

C

Here's a few general notes that I've been meaning to make for a while.

One thing to remember is that uFat simply overwrites data that is pre-allocated on the card. This needs to be a contiguous file else wierdness will ensue. One way to ensure this is to format the card before writing your proxy file to it. Deleting files from the card can leave holes in the FAT which your operating system will use next time a file is created on the card leading to a fragmented file.

Unfortunately resizing a file isn't a simple case of adjusting a value in the directory structure. It entails re-linking the FAT table chains, which means a lot more code in uFat, which is counter to its design criteria. I think it's a reasonable trade-off. Actually, thinking about it, if people were happy with yet another trade off then we might be able to come up with a small extension that can work with one or two files on a card and set their lengths in a one-off manner... But I digress.

The length of the file indicated in the directory is the raw size of the original proxy file, not the number of bytes you've written. Its size remains constant no matter how you write to the space it occupies.

If you write binary data to the file then you'll no doubt have a client application that understands it.

If you write ascii data then things are a little (but only a tiny bit) harder. When the file is written to and later removed to the computer and viewed in a text editor, there may well be a warning that the file is corrupt (at best) to a refusal to load it at all (at worst). This is to do with non-ascii ranged bytes in the file, as explained in a previous post. Warnings can be ignored on the whole. It's best practice to empty the buffer before writing data to it. You could fill it with character 0x20 (ascii for 'space') but that would mean a bloated file when editing. You could fill with 0x0d (carriage returns), which might be better. I'd go with 0 myself, unless it was making editing hard.

Thoughts encouraged.
C

Nachtwind, and those who want it, my PHP for converting to CSV:

<?php
$packet=array(3,2,2); //the number of bytes to read for each value of the 'packet' of data to be converted to a csv line.

$f=fopen('data.bin','r');
while(!feof($f)){ //repeat until end of file
      $csv_line='';
      foreach($packet AS $bytes){ // loop through the packet and generate the CSV line
            if($csv_line) $csv_line.=","; //append a comma if needed
            $csv_line.=getVal($f,$bytes); //append the next value
      }
      echo $csv_line."\n"; //echo the line
}

//reads $bytes bytes from file $f and convert them to a value
//does this by reading each bite as a character, and converting it to its ASCII value
function getVal($f,$bytes){
      $val=0;
      for($i=0;$i<$bytes;$i++){ // loop for however many bytes we require
            $b=ord(fgetc($f)); // read a character and convert it to it's ASCII byte value with ord()
            $val=($val<<8)+$b; // shift the current value left eight bits (8 bits == 1 byte) and then add the byte we just read to the value
      }
      return $val;
}
?>

You must adjust $packet to reflect the number of bytes used for each value. (This is currently configured to read the data generated from my example in SDWriter)

The magic of joining the bytes into larger values happens on this line: $val=($val<<8)+$b
this shifting of bits is how we join two bytes back into an int
this could also be achieved by multiplying by 256: $val=($val*256)+$b;

val byte2 byte1 example values and how they would be stored in multiople bytes
255 255 as we probably know, 255 is the most one byte to contain
256 1 0 to store 256 or more we need another byte to store multiples of 256. (1*256) + 0 =256
261 1 5 (1*256) + 5 =261
65535 255 255 (255*256) + 255 = 65535 //max value of an int (2 bytes)

you could now add a third byte to store multiples of 65536
this can be done for as many bytes as you need to contain your range of numbers, with only a few more bytes you can store over billions!

sirmorris, I find the functionality of uFat is excellent and the trade-offs are not a problem, especially as it seems we are all really only wanting to log data, I can't really see the need for more functionality. If I wanted to do much more I think I would probably want a full FAT filesystem. But only the future can tell :wink:

Also your method of printing to the buffer looks really cool, I haven't got time to study it greatly now, but it looks like it's exactly what JB needs and wouldn't require my suggested writeString function. In fact I guess I could extend my SDWriter in the same way to add the print/println functionality to it :smiley:

Thanks!

It looks like SDWriter would just pop right on top there!

Great stuff. I'm thrilled that uFat is finding some use.

C

I had to change about two lines and now my SDWriter supports .print() and .println() in the same way Serial does, thanks to sirmorris' genius idea :slight_smile:

The new code is available in the same place at: http://code.jhodges.co.uk/SDWriter/SDWriter.zip

There is a new example in there called Print_Data that creates a CSV style file.

The old example is still there as writing byte values is still supported.

If I may I'd like to offer a couple of comments on the writer.

Having the write function which is used by the print() calls go through a general-purpose write function is less efficient than making things work the other way viz:

void SDWriter::write(uint8_t theByte)
{
  sector_buffer[buffer_pos]= theByte;
  ++buffer_pos;
  if(buffer_pos==512)
  {
    flush();
  }
}

void SDWriter::write(unsigned long val,int bytes)
{
  for(int b=bytes-1;b>=0;--b)
  {
    write((val>>(8*b)) & 0xff);
  }
}

This way the print calls all go through the minimal version.

Good point. I guess it is the little things that count when you only have a little processor and memory :wink:

The library is updated with that change.

Thanks again.

You guys are AMAZING!! :smiley:

I've only had time this morning to quickly compile & try bobemoe's Print_Data example, but I can see that it's EXACTLY what I was after. I should be able to incorporate this code to 'format' the data stream in the most user-friendly way possible, which will make life so much easier for my workmates (and me!).

I'll be away from my computer today so I won't be putting much time into this, but in the next couple of days I'll let you know exactly how things have gone.

Charlie & bobemoe - you're work here will be extremely valuable to many others like myself I'm sure; we can't thank you guys enough.

Oh, one last thing - Charlie, did you have any thoughts on why my RTC data is being 'corrupted'?

Thanks again everyone!!

JB

I've looked at the code and there are a couple of things that I'd like to go over with you - as they're application specific maybe you should PM me and we'll take it offline via email. We can post final findings but the interim comms might be a little noisy for this topic.

Is there any hope on getting the read from SD to work?

Hi

Reading from SD is a solved problem. The discussion happening here was to do with a specific problem sketch.

Charlie

But does it read/write FAT? Will it be added to the official Arduino library page?

Hi

Kind-of, and no. The library is intended for tight memory situations where you are willing to accept limitations in usage. uFat allows you to find the starting sector and length of a file on a fat12 or fat16 formatted device.

You can then read data from and write data to those sectors. It doesn't understand the filing system more than it needs to, which in this case is only reading the boot sector, partition table and directory.

In most cases this has been quite enough for me!

Charlie

A few posts ago bobemoe posted code to use the Arduino library print routines to output strings to an mmc card. I've spoken with him and used his code as a basis for DevicePrint. It has been modified to fit my usual criteria - small & tight with few (if any) dependencies.

Get it herehttp://arduinonut.blogspot.com/2009/02/libraries.html along with some of my usual written drivel about whatever and stuff.

Commentary and questioning welcomed as usual. Enjoy.

C

wow,that looks very interestering, gonna have to finish soldering my sd card thingy ;0)

Anyway - is there a chance you could also write a function for reading stuff from an uFat File?

Hi again everyone!

After numerous emails, pulling of hair & re-writes, sirmorris has helped to get my SD card logging dreams up & running. You can get the 'final' sketch here: http://docs.google.com/Doc?id=dqhc6fg_0gmk96kdd. It's certainly not the prettiest sketch, and I can't claim much of it at all as my own work, but it does what I hoped it would (although real-world testing hasn't happened yet).

Here's a photo of my current test setup...


Needs a LOT of work before it's ready for final implementation!! ;D

Please feel free to suggest code improvements/rip my sketch to shreds - I'm no programmer, I've got thick skin, and any help is greatly appreciated!!

One last thanks to sirmorris & bobemoe for their work here - it's amazing how generous you've been with your help & knowledge.

I'll pop a note in here when I get something up in the exhibition area.

JB.