Compiling an SD/MMC and fat16 library

Hi, I'm trying to get the software library found here www.roland-riegel.de - sd-reader: MMC/SD/SDHC card library to work with the arduino. I've successfully got it compiled to object files using the same flags in the makefile that comes with the library and avr-gcc that came with arduino0010 dev environment. I placed the obj files and headers into a directory in the libraries folder so they appear as an included library. However, if I add any of the included files to a project using #include <>, it immediately blows out over the max size of the atmega168.

The code and library I am trying to get work was developed on an atmega168 so it should work. I've checked some of the other libraries and the stepper motor library is compiled to almost the same size as the sd card raw access library. I'm wondering if I've compiled it wrong or done something incorrect. The code is standard C.

Any ideas would be appreciated.

Here are a couple of random thoughts:

It may have been developed on a 168, but was it developed on a 168 with a 2k bootloader leaving only 14k of program space?

Are you linking against a library (libwhatever.a) or an object file (something.o)? If you link against the object file, everything gets stuck in the executable. If you build a library and link against it, only the functions you actually use are included in the executable.

-j

Thanks for the ideas. Yeah I compiled object files. But the object files are quite small (some are the same size as the stepper motor object files or matrix object files) so I'm wondering if I did something wrong when I compiled the code.

If I use #include <stepper.h> (or whatever it is) it uses up only a little code space. If I use #include it immediately blows out over the size limit even though the compiled object files are similar size.

I'm thinking that my object files are compiled somehow wrong and that is what is causing the problem.

Hej,

I have been experimenting with those libraries from Roland for some time now and also managed to successfully compile them. But, unfortunately, the only way of pushing them in the mega168 is getting rid of the bootloader. The hack to be made is to clean Roland's code to be much smaller or just to work at the bytes level.

I have the thing working in an Arduino board and I can write/read files and directories. With no problem, but I had to use avrdude and avrispmkII to upload the hex file ... if you have those close to you, you can try it out just for the pleasure of seeing the thing running.

/d

Thanks for that. Yeah I finally got it to all compile properly. I put the .c and .h files into a directory, renamed the .c to .cpp and it compiled automatically. I had to change a bit of the code as he uses the C99 standard and I don't know where to add flags when the arduino environment automatically compiles.

I probably will just use the low level read/write routines although the fat16 would be interesting. Just wanted to use robust sd/mmc access routines.

Hi all,

For the hardware part here is a good link (I've not tested it) http://uanr.com/sdfloppy/

Nicolas

This was a pretty ugly connection, at least with the floppy cable I tried.

Cheap SD card readers can be had for less than US$3, and every one I've looked at has a socket...

-j

hey gang,

first i looked at arduino.cc for a mmc or sd card solution, but i didn´t found something attractive for my gps logger project.
So first i tried to get a sd card solution from Ulrich Radig http://www.ulrichradig.de/home/index.php/avr/mmc-sd to work, but that was too difficult for a greenhorn like me to adapt to arduino.
After i read this thread the 10th time i gave it also a try.
Why must you "talk" so crypted? :wink: Only a fog of usable information was around! ggg Only some needed steps were explained... the ".cpp" thing and so on... but you got that to work. Would have been very helpful with a fist full of information more, but who wants a perfect world? :sunglasses:

Okay, what was interesting for me at R.Riegel sd-reader: MMC/SD/SDHC card library , that he says that you can use only the mmc part and simply leave out the fat part. http://www.roland-riegel.de/sd-reader/sd-reader_source_20071213.zip
That´s what i did, because thats enough, i only want the data stream of the gps.nmea on a BIG 1G sd card! So i can safe for sure some points of my movement profile.

For the circuit i took the simple one from Ulrich Radig, level shifting with voltage dividers, each two resistors, works like a charme!


except for the diodes for vcc i used the 3.3V of the diecimilia. there was a reset at the first mount of the board with the sd card in the slot. so i think the FTDI chip got a reset because of a high init current >50mA? At mikrocontroller.net i read a not too small elko should help, that what i´ll add next on the hardware side. May be a 100uF? any suggestions? At the distributor reichelt.de i could read at some newer sd card, that they take a current <40mA. So this could work. Otherwise i´ll take a low drop regulator.

For the arduino i made a new folder in E:\Program Files\arduino-0010\hardware\libraries , called mmc. There i placed the following files:

sd_raw.cpp  sd_raw.h  sd_raw_config.h

They got easily compiled like extensively described above here.

For the first test i used the following code:

#include <sd_raw.h>
#include <sd_raw_config.h>

int incomingByte = 0;      // for incoming serial data

void setup() {
  Serial.begin(9600);      // opens serial port, sets data rate to 9600 bps
  //Initialisierung der MMC/SD-Karte
  Serial.print("System Ready!\r\n");      
  /* setup sd card slot */

  if( !sd_raw_init()   )
  {
    Serial.println("** Keine MMC/SD Karte gefunden!! **");      

  }
  else
  {
    Serial.println("Karte gefunden!!");
  }
  struct sd_raw_info disk_info;



  Serial.print("manuf:  0x"); 
  Serial.print( disk_info.manufacturer,HEX); 
  Serial.println();
  Serial.print("oem:    "); 
  Serial.print( (char*)disk_info.oem); 
  Serial.println();
  Serial.print("prod:   "); 
  Serial.print((char*) disk_info.product); 

  Serial.print("rev:    "); 
  Serial.println(disk_info.revision); 

  Serial.print("serial: 0x"); 
  Serial.print(disk_info.serial,HEX); 
  Serial.println();
  Serial.print("date:   "); 
  Serial.print(disk_info.manufacturing_month); 
  Serial.print('/');
  Serial.print(disk_info.manufacturing_year); 
  Serial.println();
  Serial.print("size:   "); 
  Serial.print(disk_info.capacity); 
  Serial.println();
  Serial.print("copy:   "); 
  Serial.print(disk_info.flag_copy,BYTE); 
  Serial.println();
  Serial.print("wr.pr.: "); 
  Serial.print(disk_info.flag_write_protect_temp); 
  Serial.print('/');
  Serial.print(disk_info.flag_write_protect); 
  Serial.println();
  Serial.print("format: "); 
  Serial.print(disk_info.format); 


}

void loop() {

  // not used for now:
}

This works (nearly), HURRAY!

Get the following output:

Binary sketch size: 4944 bytes (of a 14336 byte maximum)

System Ready!

** Keine MMC/SD Karte gefunden!! **
manuf:  0x0
oem:    
prod:   
rev:    
serial: 0x35FF0102
date:   
size:   4027073545
copy:  
wr.pr.:

some characters are not displayable here.
Where it says, no card found ("** Keine MMC/SD Karte gefunden!! **"), i think it would need only a bit more time for the init?
The later card information seems to be no problem.

Okay, enough for today, hopefully the read write test will follow tomorrow! I pasted the dirty code in here, so others can use the information now and not sometimes. I plan also to add future additions if time allows it. There will also follow experiences with my navilock gps modul (30?, RS232 - me fool did not take the ttl one :frowning: so i had to add a max232 .... story will continue...)

cheers!

And a big THANK YOU at all the arduino gang for doing an awesome job!

I pasted the dirty code in here, so others can use the information now and not sometimes.

Yay for releasing now! :smiley:

--Phil.

Nice, its a good library and makes reading and writing to the SD card easy. There are a few parts in the sd_raw.cpp and config files that need to be changed to get reliable operation on the arduino. Something that isnt obvious is that in the library he specifies pins 5 and 6 of analog in(2 wire interface) as sensor pins for card insertion and read/write tab. You can disable that in the code but the easiest way to check if your code is working is to ground pin 5 and 6 of analog in. I found this out by accident when I was holding the arduino and tilting it and trying to figure out why it wasn't working. Every now and then it would find the card, my hand must have provided a ground. ;D

I've gone through the code and commented out various bits and changed a few things so that it works well but have to go through and make sure that all the changes work ok and document what I've changed.

Anyway yeah ground analog pin 5 and 6 and it should work.

If you want to remove the code then look in the config.h files and comment out where he defines PortC 5 and PortC 4 pins. Then remove the code that checks for memory card insertion (there's 2 functions and some code in the sd_init function) and it should work fine. If you want to leave it then you'll have to connect your memory card socket to the 2 pins for insertion and read/write checking or ground the pins.

Also if your after an SD card socket this looks nice (i found this after I'd de-soldered a memory card reader to get the SD socket lol)

Oh and just for interest, I got the FAT library compiled and running on the arduino, however I had to do quite a few changes as he uses the C99 standard and I couldn't add that flag in arduino environment. I didn't get it working fully and kept getting a few errors. I gave up in the end because it just fits into the atmega168 memory and leaves little room for anything else. You can only run a few functions before you have no room at all left. It would be good to get it running once the larger size memory atmegas are available for the arduino environment.

Much easier to write to a text file on the memory card, pull it out and chuck it in your computer to read your values! :wink:

Some short information: At the moment i have difficulties to understand really how this functions work with the "*" thing at the variables. Have to read more about C, i think. For example: uint8_t sd_raw_write_interval(uint32_t offset, uint8_t* buffer, uint16_t length, sd_raw_write_interval_handler_t callback, void* p); The Example code is extensively about the fat part, but not so for the sd_raw part.
Any help is welcome!

Also i found a forum thread, regarding to the development of this library. It´s very interesting, but sorry folks, only in german.
http://www.mikrocontroller.net/topic/48481#new
One configuration hint i found:

.....
#defines ändern: 
#define configure_pin_available() DDRC &= ~(1 << DDC4) 
#define configure_pin_locked() DDRC &= ~(1 << DDC5) 
#define get_pin_available() ((PINC >> PC4) & 0x01) 
#define get_pin_locked() ((PINC >> PC5) & 0x01) 

Wenn Dein Slot keine Pins zum Detektieren einer Karte bzw. der Position des Schreibschutzschalters hat, änderst Du diese ab zu: 
#define configure_pin_available() 
#define configure_pin_locked() 
#define get_pin_available() 0 
#define get_pin_locked() 0 
Im Makefile änderst Du folgende Zeilen passend ab: 
MCU := atmega168 MCU_AVRDUDE := m168 MCU_FREQ := 16000000UL

...

it´s about the file "sd_raw_config.h":
It means if you don´t have available-pin and locked-pin connected you should change it like in the second part of the above, with the zero at the end. The defines for the atmega168 should be fine, i think. The makefile is not of interest, i believe.
I´ll read more in that thread and will post the interesting things here.

And i´ll try that ground thing, thank you!

more to learn, more to come soon...

That define thing is handy, saves all my hacking in the library, much neater this way!

This code should write 2 bytes to the SD card starting from memory location 0. Then read them back into the tempBytes array. If you were writing your NMEA string, then read it in from serial port into an array of correct length of the string(i.e. 16 bytes or 32 or whatever an NMEA string is) then write that array to a memory location. It's pretty easy, try this example code.

// Define an array for storing data.
byte tempBytes[2];
// Store some example numbers
tempBytes[0]=23;
tempBytes[1]=57;

// Write the 2 bytes stored in tempBytes to memory location 0
sd_raw_write(0,tempBytes,2);

// Zero the tempBytes array to make sure read works ok.
tempBytes[0]=0;
tempBytes[1]=0;

// Read 2 bytes starting from memory location 0 into tempBytes print to serial output.
sd_raw_read(0,tempBytes,2)
Serial.print(tempBytes[0],DEC);
Serial.print(tempBytes[1],DEC);

@agent_orange:

that code really helped! thanks a lot! Works perfect!

At the moment i am still over it, run i a little question ... What about the 3.3V! I believe i read some time ago it´s only powered when connected to usb, got it from ftdi chip? Is that right? So i would need for sure an additional voltage regulator for external power supply.
Does someone know exactly?

Now i have arduino connected to usb with a kingston 1G card in the slot. Powered from diecimilia 3.3V supply and added a 0.47uF Elco to 3.3V power supply of the card. For the moment no problems with it.

First i will power the board with 9V alkaline for testing, later a 3.7VLiPo with stepUp. (does someone have a circuit for that? Will also add an Max1811 for LiPo charging when on usb. Have the max appnote for that.
But if someone has a working version for arduino would be fine.)

Nice sun at the moment here, hopefully get it working for a walk in the outside soon!

i will think before i post.. ;D

got the simple answer about 3.3V 2min later with measurement equipment: 3.3V pin works with a 9V alkaline.

How can i get an unsigned long int converted to it´s bytes?
I want to store the offset in the start range of the card, the first four bytes.
So this could be read the next time it starts recording and does not overwrite the record from before.

I thought code like this could help, but i googled extensively for a solution, but didn´t found something really helpful.
Here´s how i thought:

unsigned long int sdoffset=1024;  // r/w offset
// offset 0 to 1023 is for rw offset and whatever storage.
// for the first usage have to write zeroes to that.

void checkStartOffset(){
  byte offsetstored[4];
  sd_raw_read(0, offsetstored ,4);
  sdoffset=(unsigned long int)offsetstored;
  Serial.print("sdoffset read: ");
  Serial.println(sdoffset);
}
void WriteLastStartOffset(){
  char offsetstored[4];
  offsetstored=(byte)sdoffset;
  sd_raw_write(0, offsetstored ,4);
}

the start check compiles, but don´t know at the moment if that works. The read function will not compile.
Could someone point me into the right direction?

Thank you very much for any help with this!

Ok I'll try and expalin but someone else probably can do it better. If you are writing numbers larger than bytes (8bits = 2^8-1 = 255) then you need to split the number up into separate bytes. I'll give a code example. I wanted to store the values of the arduino ADC in the memory card. Values from the ADC are 10bit (10bits = 2^10-1 = 1023). So the 10bit number needs to be split into two values to be able to be stored in 8bits.

// Store my 10bit value in an int (NOT a byte, which is only 8bits) so 16 bit number. High(1111 1111) Low (1111 1111)
int temp;

byte low;    // Variable to store the low byte in
byte high;   // Variable to store the high byte in

temp=analogRead(0); //Read in the value

// AND the number with 255 which wipes the 8bits at front of number 
// The front 8 bits gets wiped->High(1111 1111) Low(1111 1111) = High(0000 0000) Low(1111 1111)
// So you can store half the number in the low byte variable
low=temp&0xFF;     


// Rotate the number 8 bits right to get the high bits.
high=temp>>8;

So you can then store each byte in an array and then write it to disc. To read back from the disc you read in each byte and then join them together again.

byte low;
byte high;
byte info[2];
int value;

sd_raw_read(0,info,2); // Read the 2 values from disc
low=info[0];  // store low byte
high=info[1]; //Store high byte
result=high<<8; // rotate left 8 bits the high value and store in result
Serial.print(result+low,DEC); //Add low and high together to get the original number you split up into 2 bytes

The easiest way to see how this all works is to try this code and use Serial.print after each operation and print the numbers in binary. Then you can see how the bit manipulations are working. For larger values long int etc, you have to split the number up into 8bytes and then store each byte individually. So if you wanted to store a 32 bit number you would need to split it into 4 bytes and then write each byte to disc.

Well, hopefully I explained it ok :slight_smile:

@agent_orange
Thanks again a lot! I think i´ve got it mostly. I made for test some handy defines:

#define HIGHBYTELONG(x)  ((byte)((x) >> 24))
#define LOWBYTELONG(x)  ((byte)((x)  >> 16))
#define HIGHBYTE(x)  ((byte)((x) >> 8))
#define LOWBYTE(x)  ((byte)((x) & 0x00ff))
#define WORD(x,y)   (( ( (byte)(x) ) << 8) | ((byte)(y)))

unsigned long int ulnumber = 0xABCDEF89;  


void setup() 
{ 
  Serial.begin(9600); 
  Serial.println(ulnumber, HEX);   // prints value as string in binary (base 2) 
  byte hbl = HIGHBYTELONG(ulnumber);
  Serial.println(hbl, HEX);   
  byte lbl = LOWBYTELONG(ulnumber);
  Serial.println(lbl, HEX);   
  byte hb = HIGHBYTE(ulnumber);
  Serial.println(hb, HEX);   
  byte lb = LOWBYTE(ulnumber);
  Serial.println(lb, HEX);   
  unsigned int wi = WORD(hbl,lbl);
  Serial.println(wi, HEX);   
  unsigned int wi2 = WORD(hb,lb);
  Serial.println(wi2, HEX);   
  unsigned long int wix = (( (((unsigned long int) wi) <<16)) | ((unsigned  int) wi2));
  Serial.println(wix, HEX);   
} 


void loop() 
{ 
}

i get the following output:

ABCDEF89
AB
CD
EF
89
ABCD
EF89
ABCDEF89

i took hex numbers, so you can see it directly if it works correct.
But i could not make a working define for the unsigned long int directly. But for now i have what i need to go on.
Thanks a lot again! :slight_smile:

Made yesterday in the late evening a little walk with the raw boxed logger device. It worked so far. But i must have had some worse serial connector or the cold was to much, @-4Celsius. Many track point missed caused by worse checksums, also the serial connection had too much failures or the mmc? . I´ll make today a walk again.
The sd_raw write works really fine now, but it is much more slower than the gps. I expected it been faster than the gps. So i changed for test the buffer to 512bytes, thats the bulk it always wants to write i think.
i´ll post that code later.

--- more info to come
added a missing "#" in code section

For an outside was no time, but tested inside. Now the data looks quite good. Saving the offset seams to work now. So the hole thing is still quite dirty, but it seems to work so far.
I take a 1G Kingston card. Data write section starts at offset 4096. The first 4 bytes on the card are for saving the actual offset.
At power on it checks the offset on the card, so it should be cleared on a fresh card or when you want to write from beginning. If it finds offset data, writing starts from there.
(On the PC i read it with HxD http://www.mh-nexus.de thats a raw reader i found for free.
You can read the data with it and clear the card if you want. Useful for test anyway.)
Then the arduino fills the write buffer with 128 bytes, the length of the serial buffer, i thought that may increase speed. Tried that with 512 bytes, i read in the sd card data sheet that this is always that bulk it writes. But that didn´t wanted to work. Went back to 128, no problem. After fill it writes the data on the card, writes the new offset and flushes the cache. Without flushing i had sometimes no data on the card.

After an 1hour test run i converted the data with GPS Babel http://www.gpsbabel.org/ to kml and checked with g**gle earth. For the test run i was on USB.
For the 3.3V i added a low drop regulator a LF33CV. I simply connected it to 5V and ground on my shield, the out pin directly to the Vin of the card. Before i had added an elco to 3.3 and ground at the card, so there was no need for an additional one. The first proposed condensator i saw no need for, cause of the stabilized 5V, but that may not be correct, but works. With my Velleman personal scope voltage looks fine without ripples, ups and downs. The Velleman may not be the best, but it´s very useful for me. Mostly better than a multimeter.

Ok, the code: Binary sketch size: 5348 bytes (of a 14336 byte maximum)
in the next post...

... continues from previous post

/* work in progress ... 2008 02 17
 * it´s still quite dirty! take care!
 *
 * using stuff from:
 * arduino.cc -> got the basics, they made it easy and fun!
 * Roland Riegel http://www.roland-riegel.de/sd-reader/ -> the awesome mmc&fat libs
 * Ulrich Radig http://www.ulrichradig.de/home/index.php/avr/mmc-sd -> hardware hint
 *                 has also made an interesting mmc & fat lib, but not used here.
 * www.mikrocontroller.net -> got some good soft-&hardware hints (german language)
 * http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1199737337 -> the important things for the code
 *                 from the nice people here to get started with this project.
 *
 * www.navilock.de -> the gps module (got the rs232, better would be the ttl level one)
 * so i had to add a max232 IC for serial communication. with ttl no need for that.
 *    http://www.navilock.de/produkte/gruppen/13/Boards_und_Module/60409_NL-500ERS_Sirf3_RS232_Modul.html
  *
 * This little project is for getting the basics of gps reading 
 * and long time data storage together.
 * Should get a useful gps-data-logger when ready.
 *
 * Ideas&ToDo:
 * buttons for r/w/a (read write append) functionality?
 * Adding a Siemens S65 LCD display.
 * Connect it to a WiiRemote -- log acceleration, degree, communicate with BT &
 *    add XBeePro for reality mesh network activity (great potential in it!)
 *    maybe for exploring land- and city scapes, reality gaming... with a group of people.
 */
// including the mmc libraries for raw r/w support:
#include <sd-reader_config.h>
#include <sd_raw.h>
#include <sd_raw_config.h>

// defines for splitting the unsigned long int into bytes for storage:
#define HIGHBYTELONG(x)  ((byte)((x) >> 24))
#define LOWBYTELONG(x)  ((byte)((x)  >> 16))
#define HIGHBYTE(x)  ((byte)((x) >> 8))
#define LOWBYTE(x)  ((byte)((x) & 0x00ff))
#define WORD(x,y)   (( ( (byte)(x) ) << 8) | ((byte)(y)))

unsigned long int sdoffset=4096;  // r/w offset
// offset 0 to 4096 is for rw offset and whatever future storage.
// for the first usage have to write zeroes to the card.
// the last offset on my 1G card is 1009254384
// get as much buffer length as the serial port has:
int in_buf_length=128;
byte incomingBuffer[128];  // for incoming serial data

// optical function control:
int ledWrite = 9; 
int ledRead = 8; 

void setup() {
  pinMode(ledWrite, OUTPUT);      // sets the digital pin as output
  pinMode(ledRead, OUTPUT);      // sets the digital pin as output

  //Init of the MMC/SD-Card
  // give it some time
  delay(500);
  digitalWrite(ledRead, HIGH);   // sets the Read-LED on
  digitalWrite(ledWrite, HIGH);   // sets the Write-LED on
  sd_raw_init ();
  delay(500);
  digitalWrite(ledRead, LOW);   // sets the Reaad-LED off
  digitalWrite(ledWrite, LOW);   // sets the Write-LED off
  delay(500);
  // and a second time...
  sd_raw_init ();
  // let´s see where we go:
  for (int i=0; i <= 3; i++){
    digitalWrite(ledRead, HIGH);   // sets the Read-LED on
    digitalWrite(ledWrite, HIGH);   // sets the Read-LED on
    delay(300);
    digitalWrite(ledRead, LOW);   // sets the Read-LED off
    digitalWrite(ledWrite, LOW);   // sets the Read-LED off
  } 
  Serial.begin(9600);      // navilock works with 9600bps from scratch
  // For programming you have to disconnect the gps physically! 
  // check where we stopped writing the last time:
  checkStartOffset();
}


void loop() {
  digitalWrite(ledRead, HIGH);   // sets the Read-LED on
  fillSerialbuffer();
  digitalWrite(ledRead, LOW);   // sets the Read-LED off
  digitalWrite(ledWrite, HIGH);   // sets the Write-LED on
  sd_raw_write( sdoffset , incomingBuffer , in_buf_length );
  sdoffset=sdoffset+in_buf_length;
  writeLastStartOffset();
  sd_raw_sync ();      
  digitalWrite(ledWrite, LOW);   // sets the Write-LED off
}


void writeLastStartOffset(){
  byte offset[4];
  offset[0]=(HIGHBYTELONG(sdoffset));
  offset[1]=(LOWBYTELONG(sdoffset));
  offset[2]=(HIGHBYTE(sdoffset));
  offset[3]=(LOWBYTE(sdoffset));

  sd_raw_write(0, offset ,4);
}
void fillSerialbuffer(){
  // we have enough storage with 1G
  // so we take the hole input from gps.
  // don´t care about single lines, get the whole stuff!
  // read one serial buffer with length of 128:
  int var = 0;
  while(var < in_buf_length-1){
    if (Serial.available() > 0) {
      incomingBuffer[var] = Serial.read();
      var++;
    }
  }
}
void checkStartOffset(){
  byte offsetstored[4];
  sd_raw_read(0, offsetstored ,4);
  unsigned int wri = WORD(offsetstored[0],offsetstored[1]);
  unsigned int wri2 = WORD(offsetstored[2],offsetstored[3]);
  unsigned long int readdoffset=( (((unsigned long int) wri) <<16)) | ((unsigned  int) wri2);
  // when it is a clean card with 00 00 00 00 on it,
  // start anyways at the right offset.
  if (readdoffset < 4096)
  {
    sdoffset=4096;
  }
  else
  {
    sdoffset=readdoffset;
  }
}

i had to cut out the out-commented debug code, it was to long.

@agent_orange
Thanks again for help! I feel having hijacked this thread, hope that is ok for everyone? Should i stay here with future additions and improvements? Because it is not purely the topic of the post now?

If there is serious interest i can describe the project with more detail, pictures and so on.

Tomorrow i will try to log my way to work. I´ll post information especially when success is not satisfying.