More than 32 bytes to microSD card?

Is there any reason why one does not seem to be able to write more than 32bytes to a txt file using the libelium microsd card module.

All of the examples only let you write 32 characters...

I presume it is possible to keep writing data to a text file until it fills the whole of the microsd card's storage space but I can't for the life of me find out how to do it...

I would like to be able to write continuously for any length of characters and then read them again.

Does anyone know how to do it?

I would also like to be able to put in new lines.

Any help is much appreciated as it is truly annoying me now...

Mowcius

Yes it is possible, I write 512 bytes to my Libelium micro sd module at a time.

I wrote a library that would take what I had buffered in my FRAM chip and then write it to the sd card in 512 byte increments.

The limit to how much you can write to a file is in the file. The sample file is only setup for a small amount of reading and writing. To increase the file simply add spaces to the file being sure to not disturb the start character. On my computer it looks like an L. Th println command is what you want to use for printing a line. print simply appends to the end of the file.

Hope that helps

Yes that does help. I will try that then. So you can only write as many characters to a text file as you have inputted blank spaces beforehand. Correct?

I presume that to write a random number of characters, you have to fill a text file with blank spaces and then just append text to it, over some of the spaces.

I have currently been fiddling with the code to get it to write to a different text file each day. I use numbers for each text file and then get it to write to the text file relating to that day. I have to have created the files first though. Unfortunately, I just managed to break my GPS which was feeding me the date data (along with other stuff) so I can't do that anymore...

So the fact is that you can only write to a file that already exists. You can only write in spaces that have already been 'filled'. You cannot physically create files and you cannot physically delete files using the arduino.

Mowcius

Yes, that is correct. I opened the file and copied the spaces and then just kept pasting them back in untill I felt the file was larg enough.

Sorry to hear about your GPS module, that sucks when things break. For testing I ended up ditching my analog input (while developing because it was easier to not have it attached) device and just read the static value of the pin. The pin is floating so it gives me small differences in results without the need to have my device attached. In your case you are actually reading the serial data (I hope thats correct) coming from your GPS module. While you are in downtime due to the GPS module I would suggest building a function that would return a mockup sample of what you would get from the GPS module so that you can keep working on the SD card code. If your not sure how to go about doing that, make up a sample output and reply to this thread and I'll try to throw together a simple function to help you out there.

Also, One thing I've noticed is that the SD card takes a while to write to. This is the reason I ended up creating a buffer to load my data in before writing to the card. This minimizes the number of times you have to write to the SD card. If the length between GPS readings is long enough than its probably not a problem and you can just write to the SD card at each reading.

Here is my code in my library that I keep talking about. This may not work (or be the best option) for you but I'm just posting it to give you an idea of what I'm talking about.

void ThrustSample::saveSample(){
      for(int i=0; i<FRAM_BUFFER_SIZE; i+=2){
            int _framBlockValue = i2c_fram_read_int(i);
            if(_framBlockValue > 0){
                  if(i == 0){
                        _sdBuffer[0] = 'T';
                        _sdBuffer[1] = secondDigitAscii(_runId);
                        _sdBuffer[2] = thirdDigitAscii(_runId);
                        _sdBuffer[3] = fourthDigitAscii(_runId);
                        _sdBuffer[4] = ',';
                        _sdBufferPos = 4;
                  }
                  else{
                        if((_sdBufferPos + 5) < 512){
                              int n = _sdBufferPos;
                              _sdBuffer[n+1] = firstDigitAscii(_framBlockValue);
                              _sdBuffer[n+2] = secondDigitAscii(_framBlockValue);
                              _sdBuffer[n+3] = thirdDigitAscii(_framBlockValue);
                              _sdBuffer[n+4] = fourthDigitAscii(_framBlockValue);
                              _sdBuffer[n+5] = ',';
                              _sdBufferPos = n+5;
                        }
                        else{
                              _sdBuffer[_sdBufferPos+1] = '\0';
                              SD.print(SAMPLES_FILE, _sdBuffer);
                              _sdBufferPos = -1;
                              delay(5);
                        }
                  }
            }
            else{
                  _sdBuffer[_sdBufferPos+1] = '\0';
                  SD.print(SAMPLES_FILE, _sdBuffer);
                  _sdBufferPos = 0;
                  break;
            }
      }
      _runId ++;
      clearBuffer();
}

Here is basically what it is doing...

  1. Create a loop the size of the memory where the readings were buffered.
  2. Read the value of the current position in memory
  3. if it's the first position create the run Id and add it to the buffer.
  4. if it's not the first position the take the integer value from the current position in memory and convert each digit of the number into a char, add a comma and store that into the buffer
  5. when the buffer gets full send that buffer to be written to the SD card
  6. As long as there is still values to be read from memory keep filling up the buffer.
  7. when there are no more values to read in memory write what remains in the buffer to the SD card
  8. increment my run id
  9. clear the buffer memory.

Well, I was getting NMEA from the GPS and then using the tinyGPS library to change it into stuff that is easier to work with. This would then give me, for example "12:24:47" (hh,mm,ss) but if any were less than 10 then I would get something like "12:3:6" rather than "12:03:06". I was using if commands to add on an extra 0 when I was writing to my serial LCD. It wouldn;t give me the values "12:24:47" though, it would give me the value of hours minutes and seconds and then I would piece them together in my code to make hh,mm,ss

I hope that made some kind of sense...

So basically whatever I wanted to save, I would have it in RAM as a value of some kind and then I could do whatever I wanted. I would like to save position data as a text file to just change into a .csv later but that requires a new line for each line of position data and I don't know if new lines can be made...

I probably won't be buying a new GPS for a bit but I would like to save some other stuff to the microSD card.

I will have a fiddle with your code and see where that gets me.

Thanks,

Mowcius

yup, that makes perfect sense.

the "SDuFAT.h" library I'm using has two print functions, print and println. These two functions will be all you need to do what you are looking for. You can combine all of the elements of a line in code and then send a println command to write the entire line at once or write each item one at a time by using the print function and then using the println function for the last element of the line.

Lets take this NMEA sample data and I'll show you what I'm saying.
"$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47"

Ok, lets say you have this line combined in code already and you want to send the entire line at once. You would simply sent the println command like this... SD.println("hola.txt","$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47");. For time sake this is going to be the faster option.

But lets say you want to send each item to the card one at a time, you could to it like this...

SD.print("hola.txt","$GPGGA,");
SD.print("hola.txt","123519,");
SD.print("hola.txt","4807.038,");
SD.print("hola.txt","N,");
SD.print("hola.txt","01131.000,");
SD.print("hola.txt","E,");
SD.print("hola.txt","1,");
SD.print("hola.txt","08,");
SD.print("hola.txt","0.9,");
SD.print("hola.txt","545.4,");
SD.print("hola.txt","M,");
SD.print("hola.txt","46.9,");
SD.print("hola.txt","M,");
SD.print("hola.txt",",");
SD.println("hola.txt","*47");

Both will render the same results. Each line will be written to the card as one line. You can change the name to a .csv and open it in excel and then write a macro to clean up the file and your done.

I created a .csv file using this sample code and just changing the .txt to .csv.

#include "SDuFAT.h"
#define MEM_PW 8

void setup(){
pinMode(MEM_PW, OUTPUT);
digitalWrite(MEM_PW, HIGH);
Serial.begin(9600);
}

void loop(){
if (Serial.available() > 0) {
int result = 0;
byte inSerByte = Serial.read();
switch (inSerByte) {
case 'R':
result = SD.cat("hola.txt");
break;
case 'P':
result = SD.println("hola.txt","$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47");
break;
}
}
}

Pardon the code, it could be cleaned up a bit but it works for a quick check.

Hope that helps.

Yes I did find the SD.println after having a bit of a look, thanks anyway...

Current issues:

[hola.txt] - deting file[hola.txt] - ..done

wtf? Typo... How do I change it? I have tried but it don't wanna work...

[hola.txt] - file info
length sectors used
9856B 1 [hola.txt] - ..done

why does it not say 9856B (or the relevant size) is used when it is full? Why does it say that it is "done"?

What are the sectors? How big does the text file have to be to take up more than 1 sector?

Also when appending data:

If I append ffffffffff. (full stop to stop appending) then I append gggg. I get ffffffffff (from the first time) and ggggffffff (from the second time). It seems to remember the last appended stuff and write it again whether or not it has been written over (in the buffer? Does it have one or something?)

Also, how do I write a full stop to a text file if it ends the appending of data...

To write to a different file can I change all of the hola.txt things to .txt and then set i as a number at the start of my sketch. Should this work if all of the numbered text files are already there?
Mowcius

wtf? Typo... How do I change it? I have tried but it don't wanna work...

I think you will need to copy the file back onto the SD card at that point.

why does it not say 9856B (or the relevant size) is used when it is full? Why does it say that it is "done"?

Because it actually is full. The file is created by filling it with a single pre-defined start character and the rest with spaces. The file and it's size has to be pre-existing. This library doesn't allow us to create a new file or change the size of any existing files.

The way it works is this... We crate a text file and make the first character a start character that was defined in the library. Then we fill the file after the start character with meaningless data. It could be a space or it could be any character you want. The space is chosen to simply make it easier on the eyes when you look at it in a text editor like notepad or something similiar. At the end of the file is the end of file characters, just as with the start character it's defined in the code. This tells the code where the file ends so that when we read the file contents the reader knows where to stop reading, otherwise it might just read the contents of the entire SD card.

It says that it's done because thats the hard coded string message that was defined by the developer of the library. It simply means that, the current operation has finished.

What are the sectors? How big does the text file have to be to take up more than 1 sector?

according to Wiki on SD cards a sector (aka a block) is 512 bytes. So once you went over 512 bytes you would be using more than one sector.

Also when appending data:

If I append ffffffffff. (full stop to stop appending) then I append gggg. I get ffffffffff (from the first time) and ggggffffff (from the second time). It seems to remember the last appended stuff and write it again whether or not it has been written over (in the buffer? Does it have one or something?)

Also, how do I write a full stop to a text file if it ends the appending of data...

Please post the line of code where you are doing this. I have an idea what's going on but I want to make sure what your doing before I explain the wrong thing.

To write to a different file can I change all of the hola.txt things to .txt and then set i as a number at the start of my sketch. Should this work if all of the numbered text files are already there?
[/quote]
You can accomplish what you are asking but not quite the way you worded it. If the file name is not going to change throughout the sketch, simply add a #define FILE_NAME "1.txt" at the top of your sketch and you should be good to go.
If you want to change the file name in code (ie... a loop) you might try the itoa function to turn the number into it's ascii value and then append the ".txt" when passing in the file name to the print function.

With regards to the typo I was talking about

deting file

This is just really annoying me and although I have looked through the sdufat library and changed any mention on deting to deleting it still says deting!

Please post the line of code where you are doing this. I have an idea what's going on but I want to make sure what your doing before I explain the wrong thing.

Ok, this is not it but this will give you an idea. Here is a rough example of what it shows me, this is from memory)

A (input A to append text)

[hola.txt] - append text

mmmmmmmmmm. (input mmmmmmmmmm.)

mmmmmmmmmm
done...

A (input A to append text again)

[hola.txt] - append text

fffff. (input fffff.)

fffffmmmmm
done...

R (input R to read data)

[hola.txt] -
mmmmmmmmmmfffffmmmmm
done...

So my serial monitor looks like:

[hola.txt] - append text
mmmmmmmmmm
done...

[hola.txt] - append text
fffffmmmmm
done...

[hola.txt -
mmmmmmmmmmfffffmmmmm
done...

So it is adding extra characters on from the last time I appended data...
And it's getting annoying >:( Grrr...

Mowcius

Ok, I don't use the append method so I'll have to take a look at it to see what it is doing.

As far as the typo, just make sure you delete the .o file after you have modified and saved the .cpp and .h files. The IDE will re-compile the files and re-create the .o file and that should take care of that issue.

Thanks
John

Ok, I will delete the .o file and try again.

Thanks!

Mowcius

Iceman, thanks for your great explanations on using SDuFat to write files.

You mentioned that it was slow to write. I'm seeing about 20 seconds / write after I made 50 or so writes. Is it that slow?

It starts out writing pretty quickly (1 sec/write) and then slows to a crawl. My "blank file" is fairly large ~130K. Is that the problem?

Any help or advice from anyone is appreciated.

I've tested it with large files without any issues. However, the first time I tried to increase the file size I accidentally removed one of the start or stop charasters (can't remember which) and it took forever to wright to it after a couple of writes.

20 seconds is way to long for one write. I'd check to make sure the file is not missing any of the special characters it needs.

Hope that helps
Thanks
John

John,
Yes you're right. The file got corrupted (again!). The 0x03 char was overwritten.

I accidentally removed one of the start or stop charasters (can't remember which)

I'd like to comment on this to confirm my understanding - which is:

  • a clean file starts with 0x03 (ETX) as the 1st byte.
  • there are 0x0A (LF) at the end of each line (of spaces) and at the end of the file (as normal)
  • as text is written, it is inserted before the ETX
  • but SD.println also appends an ETX to the line
  • so after 20 SD.println's, I see 21 ETXs in the file
  • I think the SDuFAT lib only cares about the one ETX and maybe the last LF

That's what I see when all goes well, but when using the "hola" example, it has been easy to loose the ETX, and then you're out in the ozone.

I was using Filelogger with no problems and I liked the way it didn't need a special file and the file could grow. But now I must read from the card, and I wish SDuFAT could do that too.

I am using MMC instead of SD cards (bought the wrong part!) and they seem to be more "delicate" then the SD.

For the sake of other's with problems, here are 2 observations for both types of cards:

  • If you format, or even copy, files, on a card using something like a card slot in a printer, the card may not work.
    (true for SDuFAT or FileLogger).
  • You can screw up a card to the point where it wont work even after a "proper" format.
    (I've brought them back by doing a low level format in my camera!)

I don't expect there's much to respond to in this - just wanted to get it down. I'll keep plugging (and unplugging) away with SDuFAT. For reading cards, it seems to be the only game in town.
John

Yeah, i've seen where it appends the ext char at the end but It only seems to care about the first oneas far as I can tell. I havn't had any issues with that and they end up getting treated like spaces and over written when I fill up the file.

Thanks
John

For reading cards, it seems to be the only game in town

The library for the Wave Shield also allows reading. As stated in documentation, it is based on Roland Riegel's SD library:
http://www.roland-riegel.de/sd-reader/index.html

I deleted the .o file and it re-compiled fine and worked witout the spelling mistake...

Anyone got any ideas about the issue I commented on before though. It still messes up when appending text...

Thanks,

Mowcius

I'd like to comment on this to confirm my understanding - which is:

  • a clean file starts with 0x03 (ETX) as the 1st byte.
  • there are 0x0A (LF) at the end of each line (of spaces) and at the end of the file (as normal)
  • as text is written, it is inserted before the ETX
  • but SD.println also appends an ETX to the line
  • so after 20 SD.println's, I see 21 ETXs in the file
  • I think the SDuFAT lib only cares about the one ETX and maybe the last LF

That's what I see when all goes well, but when using the "hola" example, it has been easy to loose the ETX, and then you're out in the ozone.

I've experienced the same problem. I think it happens when data
written to the file fill exactly a 512 byte sector. Here is a test
sketch:

#include <mmc.h>
#include <SDuFAT.h>
#include <microfat.h>

char myBuffer[257];

void setup (void) {
  Serial.begin (9600);

  Serial.println ("In order to run this test, hola.txt must be empty.");
  Serial.println ("You can use the hola.txt file included in SDuFAT.zip\n");

  Serial.println ("Send a character to write first block");
  while (Serial.available() == 0);
  Serial.read();
  for (int i=0; i<=256; i++) {
    myBuffer[i] = 'A';
  }
  myBuffer[256] = 0;
  int result = SD.print ("hola.txt", myBuffer);
  Serial.print ("First block written, result = ");
  Serial.println (result, DEC);

  Serial.println ("Send a character to write second block");
  while (Serial.available() == 0);
  Serial.read();
  for (int i=0; i<=256; i++) {
    myBuffer[i] = 'B';
  }
  myBuffer[256] = 0;
  result = SD.print ("hola.txt", myBuffer);
  Serial.print ("Second block written, result = ");
  Serial.println (result, DEC);

  Serial.println ("Send a character to write the final string");
  while (Serial.available() == 0);
  result = SD.print ("hola.txt", "ciao");
  Serial.print ("Final result = ");
  Serial.println (result, DEC);
}

void loop () {}

After the first print, in the file there are 256 'A' followed by 256
ETX, because SDuFAT fills the empty space of the sector with it.

After the second print, in the file there are 256 'A', followed by 256
'B', followed by spaces and a few LF. No ETX: the library "forgets" to
add it in the subsequent sector of the file.

The string "ciao" is never written to the file, because SDuFAT can't
find ETX in it.

Aurelio,
Nice job in demonstrating this bug. I got the predicted result too. I see you put an issue on Blushingboy.net. I hope when it's fixed it corrects the strange behavior I see when writing to MMC with this lib.
Thanks!

I see you put an issue on Blushingboy.net. I hope when it's fixed it corrects the strange behavior I see when writing to MMC with this lib.

In the meanwhile, there is a simple workaround that allows you to maximize the number of lines written to a file. If you write lines of fixed length l (including newline character(s)), the bug will hit when lcm(l, 512) (lcm = least common multiple: Least common multiple - Wikipedia) bytes will have been written to the file.

If you choose an odd number for l, then lcm(l, 512) = l*512: you will be able to write 512 lines to the file, which might be enough for some applications (including mine).